DOCUMENT RESUME 



ED 340 762 TM 018 018 

AUTHOR Paul, Richard W.; Nosich, Gerald M. 

TITLE A Proposal for the National Assessment of 

Higher-Order Thinking at the Community College, 
College, and University Levels. 
SPONS A312NCY National Center for Education Statistics (ED), 
Washington, DC. 
Nov 91 

53p.; Commissioned paper prepared for a workshop on 
Assessing Higher Order Thinking & Communication 
Skills in College Graduates (Washington, DC, November 
17-19, 1991), in support of National Education Goal 
V, Objective 5. For other workshop papers, see TM 018 
009-024. 

Viewpoints (Opinion/Position Papers, Essays, etc.) 
(120) — Reports - Evaluative/Feasibility (142) — 
Speeches/Conference Papers (150) 

MF01/PC03 Plus Postage. 

Cognitive Measurement; Colleges; "College Students; 
"Critical Thinhing; "Educational Assessment; 
Educational Objectives; Educational Research; 
Evaluation Methods; Higher Education; Models; 
"National Programs; "Student Evaluation; Test 
Construction; "Thinking Skills 
"National Education Goals 1990 



Conceptual foundations of a process for assessing 
higher-order thinking are reviewed, and a viable model for carrying 
out the process is presented. The first section of the paper 
formulates 21 criteria that shculd be met by any process adequate to 
the task. The second section outlines the basic concept of critical 
thinking on which the paper is based and explains how a rich and 
substantive concept of critical thinking, grounded in research, 
provides a plausible foundation for accomplishing the 21 objectives 
offered in response to the criteria. The third section explicates the 
following four domains essential to critical thinking: (1) elements 
of thought; (2) macro-abilities, or basic modes of reasoning; (3) 
traits of the mind (affective dimensions); and (4) universal 
intellectual standards. The fourth section contains recommendations 
for a process and a time-table for assessing higher order thinking 
skills at the postsecondary level. Six figures illustrate the 
discussion, and two appendices provide supplemental information. A 
20-item list of recommended readings is provided. Also provided are 
critiques by L. Boehm, P. A. Facione, and R. K. Hambleton. (SLD) 



pub date: 

NOTE 



;?UB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 
ABSTRACT 



********* »Y ************************************************************* 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
**************************************************•.******************** 



A Proposal for the National Assessment 

of Higher-Order Thinking 
at the Community College, College, and 

University Levels 

Richard W. Paul 

Director, Center for Critical Thinking 
Sonoma State University 

Gerald M. Nosich 

Assistant Director, Center for Critical Thinking 
Sonoma State University 



U.S. DEPARTMENT Of EDUCATION 
Office of Educational Raaaarch and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

n/This document he* been reproduced at 
received from tne peraon or organizetion 
originating it 

CI Minor change* heve been made to improve 
reproduction quality 



* Point* of view or opinion* stated m this docu 
ment do not necessarily represent official 
OERI position or policy 



Commissioned by: 

The United States Department of Education 
Office of Educational Research and Improvement 
National Center for Education Statistics 



©1991 Richard W. Paul and Gerald Nosich 

BEST COP Y AVAIL AILE 



Analytic Table of Contents 

Preface l 

The Problem of Lower Order Learning 1 

The State of Research into Critical Thinking and Instructional Reform 2 

The Structure of the Paper . 2 

Section One* Objectives; 21 Criteria for Higher Order Thinking Assessment 3 

Section Two. Critical Thinking and Criteria for Assessment 4 

How Does a Rich, Substantive Concept of Crit cal Thinking Meet These Criteria? 4 

A. What Is Included in a Rich, Substantive Concept of Critical Thinking? 4 

1) The National Council Definition of Critical Thinking 4 

2) Gloss on the Definition 4 

B. How Does a Rich, Substantive Concept of Critical Thinking Meet the 21 Criteria? 5 

C. What, Specifically, Are the Dangers of a Non-Substantive Concept of Critical Thinking? 9 

1) Three Serious Problems May Result 9 

2) Illustration: The California Direct Writing Assessment 9 

Section Three. The Four Domains of Critical Thinking 12 

A. Elements of Thought 12 

B. Macro-Abilities 14 

C. Affective Dimensions IS 

D. InteUectual Standards , 15 

Section Four: Recommendations of the Center for Critical Thinking 17 

A. Existing Assessment Instruments , ...17 

B. Substance and Format of Recommended Assessment Strategies 17 

1) Assessing the Domains of Critical Thinking 17 

a) Assessing Elements of Thought 18 

b) Assessing Macro-Abilities 18 

c) Assessing Affective Dimensions 19 

d) Assessing InteUectual Standards 19 

2) Varieties of Test Strategies 19 

a) Use of Multiple Choice Items 19 

b) Use of Multiple-Rating Items 21 

c) Use of Open-Ended Essay Items 24 

3) Scope of the Assessment Interdisciplinary and Intradisciplinary 24 

C. The Value of the Proposed Assessment Strategy for the Reform of Instruction 26 

D. Implementation of the Proposed Assessment Strategy , 27 

Appendix # 1; National Council for Excellence in Critical Thinking Instruction Standing Committees 29 

Appendix #2: Critique of Student Essay from CAP 30 

Recommended Readings 32 

j - 



. PREFACE 

The Problem of 
Lower Order Learning 

Virtually all informed commentators agree that school- 
ing today does not foster the "higher order thinking 
skills and abilities" which represent the "basics" of the 
future. America 2000, President Bush's education ini- 
tiative, seeks to bring schooling in line with changing 
global and economic conditions, to engender sweeping 
educational reform In what are now admittedly large- 
ly static institutions, systems highly resistant to sub- 
stantial change. America 2000 raises the following vital 
question: "How can we reverse the pervasive emphasis 
in education on lower rather than on higher order 
learning, on recall rather than on reasoning, on stu- 
dents merely "reproducing" rather than "producing" 
knowledge?" 

The state of research regarding this problem was sum- 
marized recently by Mary Kennedy in an article for 
the Kapparu 

...national assessments in virtually every subject 
indicate that although our students can per- 
form basic skills pretty well they are not doing 
well on thinking and reasoning. American stu- 
dents can compute, but they cannot reason... 
They can write complete and correct sentences, 

but they cannot prepare arguments Moreover, 

in international comparisons, American students 
are falling behind... particularly in those areas 
that require higher-order thinking. ...Our stu- 
dents are not doing well at thinking, reasoning, 
analyzing, predicting, estimating, or problem 
solving. 

In this summary. Dr. Kennedy linked the problem to the 
established mode of instruction: 

...teachers are highly likely to teach in the way 
they themselves were taught If your elemen- 
tary teache- presented mathematics to you as a 
set of procedural rules with no substantive ratio- 
nale, then you are likely to think that this is 
what mathematics is and that this is how math- 
ematics should be studied. And you are likely to 
teach it in this way. If you studied writing as a 
set of grammatical rules rather than as a way to 
organize your thoughts and to communicate 
ideas to others, then this is what you will think 
writing is, and you will probably teach it so. ... 
By the time we complete our undergraduate 
education, we have observed teachers for up to 
3,060 days 

Though not as commonly realized, this problem of the 
dominance of lower order learning is as serious in post- 
's 
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secondary as it is in primary and secondary educa- 
tion. In both undergraduate and graduate programs stu- 
dents are typically enrolled In content heavy courses 
taught by professors who feel a greater obligation to cover 
subject matter through lecture than to generate thought- 
provoking activities or assignments that may seriously 
reduce what they can cover or significantly add to their 
work load, or both. 

Alan Schoenfeld has explored this problem with respect 
to both pre secondary and post-secondary mathemat- 
ics instruction. To illustrate the detailed nature of 
what Schoenfeld's research is disclosing, here is a 
summary fr om one of his studies: 

At the University of Rochester 85% of the fresh- 
man doss takes calculus, and many goon., ./but/ 
most of these students will never apply calculus 
In any meaningful way (tf at all) tn their studies, 
or in their lives. They complete their studies with 
the Impression that they know some very sophis- 
ticated and high-powered mathematics. They 
can find the maxima of complicated functions, 
determine exponential decay, compute the vol- 
umes of surfaces of revolution, and so on. But the 
fact is that these students know barely anything 
at all The only reason they can perform with 
any degree of competency on their final exams is 
that the p'oblems on the exams are nearly carb^ 
copies of problems they have seen before; the stu- 
dents are not being asked to think, but merely to 
apply well-rehearsed schemata for specific kinds 
of tasks. Tim Ketter and I studied students' abil- 
ities to deal with pre-calculus versions of ele- 
mentary word problems.... We were not surprised 
to discover that only 19 of 120 attempts at such 
problems... yielded correct answers, or that only 
65 attempts produced answers of any kind. 

Schoenfeld summarizes the results, in general, of 
research into mathematics instruction as follows: 

In sum: all too often we focus on a narrow col- 
lection of well-defined tasks and train students 
to execute those tasks in a routine, if not algo- 
rithmic fashion. Then we test the students on 
tasks that are very close to the ones they have 
been taught If they succeed on those problems, 
we and they congratulate each other on the fact 
that they have learned some powerful mathe- 
matical techniques. In fact, they may be able to 
use such techniques mechanically while lacking 
some nidtmentary thinking skills. ToaUowthem, 
and ourselws, to believe that they 'understand' 
the mathematics is deceptive and fraudulent 

There is good reason, in our view, to link instruction- 
al reform with the need for a special emphasis on crit- 
ical thinking, problem solving, and communication 
skills, for it is precisely these higher order thinking skills 
that are routinely sacrificed when coverage and lower 
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order recall dominate the classroom at either the pre- 
or post-secondary level, as they now do. 

The State of Research Into 
Critical Thinking and 
Instructional Reform 

One major value of the last ten years' of research into 
critical thinking is the focus on the need for reform of 
instruction at all levels: on the need for students to rea- 
son mathematically in mathematics courses, to reason 
historically in history courses, to reason scientifically 
in science courses, to reason sociologically in sociolo- 
gy courses. Indeed, critical thinking research has 
emphasized three basic needs for all learning: for all stu- 
dents to reason out all basic concepts and under- 
standings, to reason to all basic conclusions and solu- 
tions, and to reason through and across the 
curriculum. 

This emphasis has been embedded to the structure of 
the 1 1 major international conferences on research 
into critical thinking and educational reform (1980- 
1991) held at Sonoma State University, the last attract- 
ing 1400 registrants from 20 countries and involving 
over 350 sessions representative of a wide variety of aca- 
demic disciplines. This same emphasis is reflected to 
the 25 or so other conferences focused on critical 
thinking in the last ten years (at Harvard, the University 
of Chicago, Montclair State, Oakton College, and else- 
where), and to most of the articles published concern- 
ing critical thinking. 

What is more, the research into critical thinking has 
focused not only on the cultivation of reasoning in all dis- 
ciplines but also on generallzable standards for the 
assessment of reasoning as well. The concepts and dis- 
tinctions embedded to critical thinking research are, as 
a result, well-suited for the design of a process to assess 
higher order thinking. In this paper we shall set out both 
the conceptual foundations for such a process as well 
as a viable model for carrying out that process. 

Before we spell out the detailed structure of this paper, 
however, it is Important to note that the concept of crit- 
ical thinking has not played a central role to the design 
of educational assessment Instruments to date princi- 
pally because the concept has been developed exten- 
sively only over the last ten years, and therefore has not 
had time to permeate already developed assessment 
tools. Now that we possess a rich, substantive concept, 
however, we have an unprecedented opportunity to 
assess central rather than peripheral aspects of criti- 
cal thinking, and to do so to an authentic and repre- 
sentative way. If anything less than this concept and its 
central aspects is assessed, the ultimate goal of fostering 
higher order thinking as an academic, social, and voca- 
tional need will be ill served. 



The Structure of the Paper 

The substance of this paper is divided into four sections, 
each focused on a major question, as follows: 

Section One 

What should be the main objectives of a process to 
assess higher-order thinking at the post-secondary 
level? 

Section Two 

How does a rich, substantive concept of critical think- 
ing meet these criteria? 

(A) What Is included in a rich, substantive concept 
of critical thinking? 

(B) How, specifically, does this concept meet the 
criteria? 

(C) What, specifically, are the dangers of a non- 
substantive concept of critical thinking? 

Section Three 

What are the four component domains of critical think- 
ing and the implications of each of these domains for 
the assessment of higher -order thinking? 

Section Four 

What is the simplest solution to the design of a process 
to assess higher-order thinking at the post-secondary 
level, given the answers to questions one through three 
above? 

The first section of the paper formulates 21 objectives 
that should be met by any process adequate to the 
task. The second outlines the basic concept of critical 
thinking which Informs the paper and explains how a 
rich, substantive concept of critical thinking, grounded 
to the research on critical thinking, provides a plausi- 
ble foundation for accomplishing these objectives. The 
third section of the paper explicates the four domains 
essential to critical thinking: 

A) The Elements of Thought (eight essential 
dimensions of all reasoning crucial for 
understanding and assessing reasoning), 

B) Macro-Abilities (basic modes of reasoning- 
Including reading, writing, speaking, and 
listening— that represent modal 
"orchestrations" of the elements 

of thought). 

C) Traits of Mind (the affective support without 
which critical thinking skills are merely 
episodically used, and often to a limiting 
rather than an expansive manner), and 



2 



D) Universal intellectual Standards 

(presupposed by critical thinking) 

As we give a brief explication of the elements of thought, 
the macro- abilities, and essential traits of mind, we 
briefly comment on the implications for assessment pur- 
poses of each conception. 

In the fourth and final section of the paper, we lay out 
our recommendations for a process and a time-table for 
assessing higher order thinking skills at the post-sec- 
ondary level. 



SECTION ONE 

OBJECTIVES 

What should he the main objectives 
of a process to assess higher order 
thinking at the post-secondary 
level? 

1) It should assess students' skills and abilities in 
analyzing, synthesizing, applying, and 
evaluating information. 

2) It should concentrate on thinking skills that 
can be employed with maximum flexibility, in a 
wide variety of disciplines, situations, contexts. 

3) It should account for both the important 
differences among disciplines and the skills, 
processes, and affective dispositions that are 
crucial to all the disciplines. 

4) It should focus on fundamental, enduring 
forms of intellectual ability that are both fitted 
to the accelerating pace of change and deeply 
embedded in the history of the advancement of 
the disciplines. 

5) It should readily lead to the Improvement of 
instruction. 

6) It should make clear the intercomiectedness of 
our knowledge and abilities, and why expertise 
in one area cannot be divorced either from 
findings in other areas or from a sensitivity to 
the need for interdisciplinary integration. 

7) It should assess those versatile and 
fundamental skills that are essential to being a 
responsible, decision-making member of the 
workplace. 

8) It should be based on clear concepts and have 
well-thought-out, rationally articulated goals, 
criteria, and standards. 

9) It should account for the integration of adult- 
level communication skills, problem-solving, 



and critical thinking, and it should assess all of 
them without compromising essential features 
of any of them. 

10) It should respect cultural diversity by focusing 
on the common-core skills, abilities and traits 
useful in all cultures. 

1 1) It should test for thinking that is empowering 
and that therefore, when incorporated into 
Instruction, promotes (to quote the September, 
1991 Kappan) "the active engagement of 
students in constructing their own knowledge 
and understanding." 

12) It should concentrate on assessing the 
fundamental cognitive structures of 
communication at the college-level, for 
example: 

with reading or listening, the ability to 

• create an accurate interpretation, 

• assess the author's or speaker's 
purpose, 

• accurately identify the question-at-lssue 
or problem being discussed, 

• accurately identify basic concepts at the 
heart of what is said or written. 

• see significant implications of the 
advocated position, 

• identify, understand, and evaluate the 
assumptions underlying someone's 
position, 

• recognize evidence, argument. Inference 
(or their lack) in oral and written 
presentations, 

• reasonably assess the credibility of an 
author or speaker, 

• accurately grasp the point of view of the 
author or speaker, 

• empatheUcally reason within the point 
of view of the author or speaker. 

with writing and speaking, the ability to 

• identify and explicate one's own point of 
view and its implications, 

• be clear about and communicate 
clearly, in either spoken or written form, 
the problem one is addressing, 

• be clear about what one is assuming, 
presupposing, or taking for granted, 

• present one's position precisely, 
accurately, completely, and give rele- 
vant, logical, and fair arguments for it, 

• cite relevant evidence and experiences 
to support one's position, 
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• see, formulate and take account of 
alternative positions and opposing 
points of view, recognizing and 
evaluating evidence and key 
assumptions on both sides, 

• illustrate one's central concepts with 
significant examples and show how they 
apply to real situations, etc., 

• empathetlcalry entertain strong 
objections from points of view other than 
one's own. 

13) It should assess the skills, abilities and 
attitudes that are central to making sound 
decisions and acting on them to the context of 
understanding our rights and responsibilities 
as citizens, as well-informed and thinking 
consumers, and as participants to a symbiotic 
world economy. 

14) It should avoid any reductionlsm that allows a 
multi-faceted, theoretically complex, and 
authentically usable body of abilities and 
dispositions to be assessed by means of 
oversimplified pans that do not adequately 
reflect the v/hole. 

15) It should enable educators to see what kinds 
of skills are basic at the college level. 

16) It should be of a kind that will assess valuable 
skills applied to genuine problems as seen by a 
large body of the populace both inside and 
outside of the university community. 

17) It should include items that assess both skills 
of thoughtfully choosing the most reasonable 
answer to a problem from among a pre-selected 
set and also the skills of formulating the 
problem itself and of making the initial 
selection of relevant alternatives. 

18) It should contain items that, as much as 
possible, are examples of the real-life problems 
and Issues that people will have to think out 
and act upon. 

19) It should allow a financially affordable means 
of assessment. 

20) It should enable colleges to assess the gains 
they are making to teaching higher-order 
thinking. 

21) It should provide for a measure of achievement 
against national standards. 



SECTION TWO 

CRITICAL THINKING AND 
CRITERIA FOR ASSESSMENT 

How does a rich, substantive 
concept of critical thinking 
meet these criteria? 

A. What is included in a 
rich, substantive concept of 
critical thinking? 

Most of the language we shall use is drawn from draft 
statements of the National Council For Excellence In 
Critical Thinking Instruction. The National Council 
has been established precisely to articulate standards 
to critical thinking by 50 key leaders in critical think- 
ing research and 105 leading educators. It is to process 
of establishing 8 regional offices and setting up 75 
research-ba'jed committees to articulate the state of 
research to the field. (See Appendix #1.) 

NATIONAL COUNCIL DEFINITION 
"Critical thinking is the intellectually disciplined process 
of actively and skillfully conceptualizing, applying, ana- 
lyzing, synthesizing or evaluating information gathered 
from, or generated by, observation, experience, reflec- 
tion, reasoning, or communication, as a guide to belief 
and action." 

This Is the working definition of the National Council 
for Excellence to Critical Thinking Instruction. Though 
the definition as well as the other draft statements of 
the Council are subject to modification and refine- 
ment, the basic idea is one that is common to practi- 
tioners and researchers to critical thinking. 

GLOSS ON THE DEFINITION 
"In its exemplary form, (critical thinking) is based on 
universal intellectual values that transcend subject-mat- 
ter divisions: clarity, accuracy, precision, consistency, 
relevance, sound evidence, good reasons, depth, 
breadth, and fairness" (National Council Draft 
Statement). 

(a) "It entails the examination of those structures 
or elements of thought implicit to all reasoning: 
purpose: problem, or question-at-lssue; 
assumptions: concepts: empirical grounding: 
inferences: implications and consequences: 
objections from alternative viewpoints, and 
frame of reference'* (National Council Draft 
Statement). 
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(b) It entails larger-scale abilities of integrating 
fc elementary skills in such a way as to be able 
to apply, synthesize, analyze, and evaluate 
complicated and multidimensional Issues. 
These Include such macro-abilities as clarifying 
issues, transferring insights into new contexts, 
analyzing arguments, questioning deeply, 
developing criteria for evaluation, and 
assessing solutions, refining generalizations, 
and evaluating the credibility of sources of 
information Among the macro-abilities are 
included also the central forms of 
communication: critical reading, writing, 
speaking, and listening. Each of them is a 
large-scaled mode of thinking which is 
successful to the extent that it Is informed, 
disciplined and guided by critical thought and 
reflection (paraphrased from National Council 
Draft Statement). 

c) Critical thinking entails the possession and 
active use of a set of traits of mind and affective 
dimensions: independence of thought, 
falrmindedness. Intellectual humility, 
intellectual courage, intellectual perseverance, 
intellectual Integrity, curiosity, confidence in 
reason, the willingness to see objections, to 
enter sympathetically into another's point of 
view, to recognize one's own egocentricity or 
ethnocentrlclty. (paraphrased from National 
Council Draft Statement). 

"Critical thinking—in being responsive to variable sub- 
ject-matter, issues, and purposes— is incorporated In a 
family of Interrelated modes of thinking, among them: 
scientific thinking, mathematical thinking, historical 
thinking, anthropological thinking, economic think- 
ing, moral thinking, and philosophical thinking" 
(National Council Draft Statement). 

B. How does a rich, substantive 
concept of critical thinking meet the 
21 criteria? 

In our view, a rich, substantive concept of critical 
thinking, and it alone, provides an intelligible and 
workable means of meeting all 21 criteria. In this sec- 
tion we will briefly consider each objective in turn, not 
as a definitive response to the criteria, but merely to 
suggest the f iller response in Section Three below. 

CRITERION • 1 
Can it be used to test Information processing skills? 

Critical thinking Includes at its core "a set of informa- 
tion and belief generating and processing skills and abil- 
ities." 



CRITERION §2 
Can it be used to test flexible skills and abilities that 
can be used in a wide variety of disciplines, situa- 
tions, and contexts? Since the art of critical thinking 
"entails proficiency in the examination of those struc- 
tures or elements of thought Implicit in all reason- 
ing—purpose, problem or questlon-at-lssue. assump- 
tions, concepts, empirical grounding, reasoning leading 
to conclusions, implications and consequences, objec- 
tions from alternative viewpoints, and frame of refer- 
ence"— it provides for maximum flexibility of use. It 
can be used in any discipline, with respect to any sit- 
uation to be figured out. any context in which reason- 
ing is germane. 

CRITERION # 3 
Can It account for important differences among 
the disciplines? Disciplines differ not because some 
make assumptions and others do not. not because 
some pose questions or problems and others do not. not 
because some have purposes and others do not. but 
rather because each has somewhat different purposes, 
and hence asks somewhat different questions and 
poses somewhat different problems and gathers some- 
what different evidence and uses somewhat different 
concepts, etc... Critical thinking highlights these dif- 
ferences while yet underlining common structural fea- 
tures. 

CRITERION #4 
Can It be used to focus on fundamental abilities fit- 
ted to the accelerating pace of change and embed- 
ded In Intellectual history? Basic critical thinking 
skills and abilities are readily shown to be implicit in 
the ration* < development and critique of ideas at the 
core of intellectual history. They explain, for example, 
how new disciplines emerge from established ones: 
that is. by asking new questions, pursuing new pur- 
poses, framing new concepts, gathering new data, 
making new assumptions, reasoning in new direc- 
tions, etc.. .They explain as well how it Is that a new field 
of study can ground itself, even at the outset, on defi- 
nite intellectual standards that transcend any partic- 
ular academic field: clarity, precision, accuracy, rele- 
vance, consistency, evidentiary force, valid reasoning, 
consistency . . . (standards implicit in the history of crit- 
ical thinking and rational discourse, in every domain). 

CRITERION # 0 
Can It be nsed to lead to the Improvement of 
Instruction? Critical thinking is not an Isolated good 
unrelated to other Important goals in education. Rather 
it is a seminal goal which, done well, simultaneously 
facilitates a rainbow of other ends. It is best conceived, 
therefore, as the hub around which all other educational 
ends cluster. For example, as students learn to think 
more critically they become more effective readers. 
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writers, speakers, and listeners (because all require well- 
reasoned thought). They increase their mastery of con- 
tent (because all content is embedded in a system of 
understandings which, to be grasped, must be reasoned 
through). They become more proficient in— because 
they must be practiced within— a variety of modes of 
thinking: for example, historical, scientific, and math- 
ematical thinking. Self-confidence increases with the 
intellectual empowerment critical thinking engenders. 
Finally, they develop skills, abilities, and traits of mind 
(intellectual discipline, intellectual perseverance, intel- 
lectual humility. Intellectual empathy, intellectual 
integrity.....) crucial to success to the professional and 
everyday world. 

CRITERION #6: 
Can It make clear the Interconnectedness of our 
knowledge end abilities, and why expertise In one 
area cannot be divorced either from findings In 
other areas or from a sensitivity to the need for 
taterdlsdpllnary Integration? In learning to think crit- 
ically one learns to transfer what one has learned 
about the logic of questions to one field to logically sim- 
ilar questions in other fields. Typically this begins with 
a recognition of the need to ask questions based on log- 
ical parallels between all fields of study, for example, 
skilled practice to questioning concepts and theories, 
to questioning data, to questioning the source or inter- 
pretation of data, to questioning the nature or organi- 
zation of data, to questioning inferences, to question- 
ing assumptions, to questioning implications and 
consequences, and to questioning points of view and 
frames of reference, etc. 

CRITERION* 7 
Can It be used to essess those versatile and funda- 
mental skills that ere essential to being a respon- 
sible, decision-making member of the workplace? 

Critical thinking skills and abilities are highly trans- 
ferable to the workplace. Since to learning to think crit- 
ically we learn to take increasing charge of our mind as 
an instrument of learning— for example, reading, writ- 
ing, speaking, and listening with greater discipline and 
skill-— we are well situated to engage to collective prob- 
lem solving and goal attainment, wherever they occur. 
The kind of "work" Increasingly required to industry and 
business is "intellectual", i.e.. requiring that workers 
define goals and purposes clearly, seek out and orga- 
nize relevant data, conceptualize those data, reason to 
legitimate conclusions, consider alternative perspectives, 
adjust thinking to context, question assumptions and 
modify thinking in the light of the conttoual influx of 
new information. Furthermore, the intellectual work 
required must increasingly be coordinated with, and 
must profit from the critique of fellow workers. There 
is no avoiding the need, therefore, to express ideas 
well, accurately represent and consider fairly the ideas 
of others, write clear and precise memos and docu- 

ERJC 



ments, and coordinate and sequence all of these so that 
well-reasoned policies and decisions can be accurate- 
ly understood and effectively implemented. 

CRITERION* 8 
Can it generate clear conce pts and well-thought-out, 
rationally articulated goals, criteria, and standards? 

Since critical thinking is based on the art of monitor- 
ing one's thinking with standards implicit to the uni- 
versal structure of thought and since the use of these 
standards with respect to the structure of thought is 
implicit to intellectual history from Socrates through 
Einstein, there is no problem using critical thinking to 
generate clear concepts for testing, as well as rationally 
articulated goals, criteria, and standards. 

CRITERION* 9 
Cm It account for the Integration of adult-level 
communication skills, problem-solving, and critical 
thinking, and legitimately assess all of them with- 
out compromising essential features of any of them? 
Shallow concepts of critical thinking often distinguish 
critical thinking from problem solving and decision 
making as well as from reading, writing, and speaking 
skills. Once one considers a rich, substantive concept 
of critical thinking, however, it is clear that each of the 
basic skills of critical thinking are presupposed by 
each of the other skills. Just as each of them Is deeply 
interrelated to critical thinking as a whole. Consider, 
does it make sense to analyze potential solutions to 
problems or the implications of choosing an alternative 
to making a decision without using critical thinking? 
Clearly not. In the first place, every problem can be 
expressed to the form of one or more questions one is 
attempting to settle. Every problem to be solved (or 
question to be settled) requires a critical analysis of the 
conditions under which it can be solved or settled. 
We, as problem-solvers, need to look critically at the 
purpose for which we are attempting to settle the ques- 
tion, we need to critically examine contextual factors, 
our assumptions, our concepts, what we are using as 
data, our organization of the data, the source of the 
data, our reasoning, the Implications of our reasoning, 
our point of view, objections from other points of view. 
All of these are essential to higher order problem solv- 
ing and decision making. Furthermore, all of these 
intellectual abilities are crucial to higher order reading, 
writing, speaking, and listening. Reading requires that 
we analyze the text and re-create its logic to our own 
minds. Writing requires that we construct a logic that 
can be readily translated into the logic of the thinking 
c* our potential readers. Speaking requires that we 
articulate our thoughts to such a way that those who 
are listening can translate our thoughts into their 
experiences. And listening requires that we analyze 
the logic of the thinking of the speaker. Intellectually dis- 
ciplined reading, writing, speaking, and listening 
require, to other words, that we work explicitly with the 
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logic we are constructing or re-constructing, using our 
grasp of the standards of critical thinking to commu- 
nicate accurately and precisely, effectively solve prob- 
lems, and rationally make decisions. 

CRimnorf 10 

Does It respect cultural diversity by focusing on 
the common-core skills, abilities and traits useful in 
all cultures? As the criterion presupposes, we can 
respect cultural diversity best by constructing tests in 
higher order thinking that focus on skills and abilities 
necessary in all modern cultures. In Mils way we can 
legitimately Justify assessing it in all cultural groups. 
Basic critical thinking skills and abilities— because they 
are based on fundamental elements implicit in the 
structure of all reasoned thought per sc. and because 
their mastery is essential to higher order thinking in all 
academic, professional, personal and public life— are an 
appropriate foundation for assessment. 

CtUtEKIONiU 
Does it test lor thinking that promotes (to quote the 
September, 1991 Kappan) "the active engagement 
of students In constructing their own knowledge 
and understanding?'' Narrow concepts of critical think- 
ing sometimes characterize it in negative terms, as a set 
of tools for deciding if we are making mistakes in think- 
ing. A rich, substantive concept of critical thinking, 
however, highlights its central role In all rationally 
defensible thinking, whether that thinking is focused on 
assessing thought or products already produced or 
actively engaged in the construction of new knowledge 
or understandings. Well-reasoned thinking, whatever its 
end, is a form of creation and construction. It devises 
and articulates purposes and goals, translates those 
goals into problems or questions, seeks data that bear 
upon problems or questions, interprets those data on the 
basis of concepts and assumptions, and reasons to 
conclusions within some point of view. All of these are 
necessary acts of the reasoning mind and must be 
done "critically" to be done well. Hence all require opt- 
ical thinking. 

CRITERION # 12 
Does it concentrate on assessing the fundamental 
cognitive structures of communication at the col- 
lege-level? Each of the dimensions Identified ii the 
objective Is either straightforwardly a critical thinking 
ability or dependent on a critical thinking ability. The 
writer's or speaker's purpose, Implications, assump- 
tions, point of view, etc., are all elements of thought, 
and the ability to Identify and assess those as one 
reads or listens— the ability to construct in one's mind 
an accurate and fertile Interpretation— are simply 
modes of thinking by listening, winking by reading. 

A similar reliance on elements of thought is central to 
writing or speaking effectively at the post-secondary 
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level. The knowledge of how to amass evidence, to 
make clear one's own assumptions, to see the impli- 
cations of a position: these are critical thinking 
macroablllties. 

All forms of communication, moreover, rely on critical 
thinking standards. Essays and interpretations of 
essays, utterances and interpretations of utterances, 
need to be relevant, logical, conslstentiy worked out; 
evidence needs to be recorded and reported accurate- 
ly; points need to be made clearly and with as much 
precision as the subject permits; topics need to be 
covered in depth and presented fairly. 

CRITERION § 13 
Can It be used to assess the central features of 
making rational decisions as a citizen, a consumer; 
and a part of a world economy? Both public and pri- 
vate life increasingly require mastery of the basic skills 
and abilities of critical thinking. When this mastery is 
absent the public degenerates Into a mass society sus- 
ceptible to manipulation by public relations specialists 
who can engineer political victories by an adroit use of 
mud slinging, scare tactics, shallow nationalism, fear, 
envy, stereotype, greed, false idealism, and maudlin sen- 
timentalism Modern citizenship requires basic critical 
thinking skills and abilities throughout The modern cit- 
izen should be able to assess the arguments present- 
ed for his or her assent, must rationally adjudicate 
between conflicting points of view, must attempt to 
understand a culturally complex world, must assess the 
credibility of diverse sources of Information, must 
translate between conflicting points of view and diverse 
appeals, must rationally decide between priorities, 
must seek to understand complex issues th involve 
multiple domains (for example, the environmental, 
moral, economic, political, scientific, social, and his- 
torical domains). Without a solid grounding to the 
skills and abilities of critical thinking, citizens are 
intellectually disarmed, incapable of discharging their 
civic responsibilities or rationally exercising their rights. 

CJoT£RJ0N#14 
Can It avoid the reductionism of a complex whole 
to oversimplified parts? Testing for a rich, substan- 
tive concept of critical thinking is testing for skills of rea- 
soning to terms of elements of thought, for macro- 
abilities that are orchestrations of those elementary 
skills, for the affective dimensions that make critical 
thinking actuallzable in practice, and for universal 
intellectual standards, to short for a rich and complex 
whole rather than for fragmented parts. 

CRITERION §15 
Can It articulate what Is central to college-level 
basic skills? Basic skills at the college-level are con- 
stituted by the structures explicated In a rich, sub- 
stantive concept of critical thinking. To teach reading 
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at the college-level is to teach the ability not merely to 
repeat content, but to reconceptuallze that content, to 
see applications -jf the main ideas, to generalize from 
them, critique them, see them in context, to enter with 
empathy Into another's point of view. To teach writing 
as a basic skill at the college-level is to teach not mere- 
ly grammar and punctuation, but the ability to arrange 
one's ideas logically and consistently, to anticipate 
reasonable objections, to transfer Ideas to the page In 
a way that makes them decipherable in all their com- 
plexity by a reader. To teach math as a basic skill at the 
college-level is not primarily to teach how to solve pre- 
selected. Individual, isolated problems out of context, 
but to teach the ability to begin to make sense of the 
world mathematically, to think quantitatively, to be 
able to see mathematical patterns, to set up the con- 
struction of problems and then creatively go about 
solving them. Critical-thinking abilities like these do not 
exist somehow In addition to the basic skills of college 
work; they constitute the basic skills of college work. 

CRITERION §16 
Can it provide the kind of skills that are seen as valu- 
able outside the university as well as Inside It? 

Critical thinking provides skills that are seen as valu- 
able by practitioners of the academic disciplines, by 
responsible leaders of government, of the professions, 
of business, by citizens interested in their environ- 
mental, physical and economic welfare. In all such 
areas what Is needed are ways to adapt to rapidly 
changing knowledge, to recognize problems and see 
their implications before they become acute, to formu- 
late approaches to their solution that recognize legiti- 
mately different points of view, to draw reasonable con- 
clusions about what to do. Increasingly, one is hearing 
statements such as the one made by David Kennedy, the 
president of Stanford University, to 3000 college and uni- 
versity presidents: 

"It simply will not do for our schools to produce 
a small elite to power our scientific establishment 
and a larger cadre of workers with basic skills 
to do routine work. Millions of people around 
the world now have these same basic skills and 
are willing to work twice as long for as little as 
one-tenth our basic wages. To maintain and 
enhance our quality of lye, we must develop a 
leading-edge economy based on workers who 
can think for a living. If skills are equal in the 
long run wages will be too. This means we have 
to educate a vast mass of people capable of 
thinking critically, creatii ty, and imaginatively. m 

CRITERIA #17 AND §18 
Can critical thinking be assessed In a way that 
requires evaluation of authentic problems In realistic 
contexts, where the abilities assessed Include those 
of formulating the problem and Initial screening of 
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plausible solutions? Yes. Testing of authentic skills, 
abilities and dispositions in authentic contexts can be 
accomplished by using a combination of (a) standard 
multiple-choice items, (b) machlne-gradable multiple- 
rating items and (c) short essay items. 

(a) The standard multiple-choice part of the 
assessment would be an expanded version of 
established critical thinking tests, such as the 
Watson-Glaser or Cornell tests. This would test 
the ability to select, from among a sample, the 
most reasonable alternative. It Is suitable for 
assessing micro-dimensional critical thinking 
skills, like identifying the most plausible 
assumption, recognizing an author's purpose, 
selecting the most defensible inferences, and 
such like. 

(b) The multiple-rating part of the assessment 
would test more open-ended and larger - 
domained abilities, like thinking within 
opposing points of view, the willingness to 
suspend Judgment, the ability to synthesize 
disparate data into a logical scheme, taking 
established findings and generalizing them into 
new contexts, etc. 

The multiple-rating portion of the assessment, for it to 
be reliable, must 

I) embody a rich and substantive idea of 
critical winking. 

II) be composed and monitored by critical 
thinking experts who have such a 
concept. 

III) be changed often (5% annually) to 
assess critical thinking with respect to 
authentic contemporary issues. 

(c) The essay part of the assessment would be 
designed to address critical thinking abilities 
and traits that involve creating a logic to 
capture a situation rather than selecting from 
among possibilities suggested by the test. 
Examples include the ability to construct an 
Interpretation, to make a logical outline of a 
text, to figure out ways to gather information, 
to take an unclear and complex real Issue and 
reformulate it so as to make it more amenable 
to solution. 

Validity on the essay part of the assessment requires 
that the test be 

i) composed by experts in critical 
thinking. 

11) assembled from a large and rotating 
bank of short essay questions to allow 
for items that show no significant 
differences. 
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ill) centrally graded by teams well-trained 
in a full concept of critical thinking in 
order to assure quality control. 

CRITERION # 19 
Can critical thinking at the post-secondary level be 
assessed nationally In n way that Is financially 

affordable? To make it affordable, the constructed 
response segment of the assessment should be admin- 
istered not to the population of students as a whole, but 
rather to a representative sample of the student pop- 
ulation of a college or university. The assessment 
should be (a) paid for by colleges and universities that 
contract to have their students tested, and (b) con- 
structed, monitored, administered and graded by a 
private agency with critical thinking credentials, or at 
least under the direction of scholars with a solid 
grounding in the research into critical thinking. 

CRITERIA §20 AND §21 
Can critical thinking be assessed so as to gauge the 
Improvement of their students over ti?e course of 
their college education and to measure the achieve- 
ment of their students against national standards? 
To evaluate students in both these dimensions requires: 

i) an assessment administered as a pre-test 
before university-entrance, at the end of the 
second year, and Just prior to graduation (to 
provide for value-added Judgments). 

il) a criterion-referenced assessment that is built 
on clear, consistently applied quality-norms 
that are derived from a rich and substantive 
concept of critical thinking (to provide for the 
measuring of national progress). 

C. What, Specifically, Are the 
Dangers of a Non-Substantive 
Concept of Critical Thinking? 

It is Important to be alert to the dangers posed by a non- 
substantive concept of critical thinking. Such a concept 
exists when, separate from a consideration of the 
research in the field, a person or institution presupposes 
(a) that the meaning or terminology of critical thinking 
is intuitively obvious (hence not to need of scholarly 
analysis), or (b) that each concept underlying critical 
thinking (such as assumption, inference, implication, 
reasoning,...) can be analyzed separately from a theo- 
ry that accounts for the interrelation of these con- 
cepts, or (c) that the skills of critical thinking can be ade- 
quately cultivated without reference to the values, 
traits of mind, and dispositions that underlie those 
skills. 

1) There are at least three serious problems that may 
result from the use of a theoretically superficial con- 
cept of critical thinking: 
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1) important critical thinking concepts, which 
must be clearly defined to be used effectively to 
assessment, may be used vaguely, 
inconsistently, incorrectly, or rnlsleadlngly, 

2) a false, misleading, or simplistic over-arching 
concept of critical thinking may be fostered, 
and/or 

3) an unrealistic strategy for the assessment and 
cultivation of critical thinking may be 
incorporated into testing and teaching. 

Many examples of the unwitting use of a non-substan- 
tive concept of critical thinking could be cited— such as 
'thinking skills" programs devoid of Intellectual stan- 
dards (which, for example, systematically confuse "infer- 
ences" with "valid inferences" and "analogies" with 
"sound analogies"), or testing personnel who lack ade- 
quate grounding to critical thinking theory (and so, for 
example, frequently confuse assumptions with inferences 
or inferences with implications). The most far-reaching 
danger occurs when influential educational systems or 
institutions, like state departments of education, inad- 
vertently incorporate a non-substantive concept of crit- 
ical thinking into statewide curriculum standards or into 
statewide testing programs. This can result in significant, 
unintended negative consequences, for example: thou- 
sands of teachers encouraged to follow a misconceived 
model for the assessment of reasoning, leading to misto- 
structlon on a grand scale. 

2) illustration We shah look at one important case. 
Unfortunately, given the brevity of this paper, one case 
must stand for all. The case we have chosen concerns 
the Integrated Language Art* Assessment of the 
California Assessment Program, a massive statewide 
program that has impact not only on every student to 
the public schools of California, but also, because of the 
leadership role of California in assessment, on nation- 
al teaching and testing practices as wen. It appears that 
three fundamental mistakes occurred to the design of 
the direct writing assessment: 

1) Though one of the goals of the program was to 
place an emphasis on quality of reasoning and 
critical thinking to writing, it appears tuat no 
one with a research background to critical 
thinking reviewed the articulation or 
implementation of the assessment prompts (We 
infer this from the fact that fundamental 
conceptual errors occur both to the prompts 
themselves and to the application of criteria to 
student constructed responses.) 

2) It was assumed, inappropriately, that 
classroom teachers without extend -d training 
in critical thinking are able to effectively assess 
student assays that call for evaluative 
reasoning. (We infer this from statements 
descriptive of the assessment design like: 
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"Teachers on the CAP writing Development 
Team develop all the testing and Instructional 
* materials for assessment. For every type of 
wilting assessed, the team develops a special 
set of prompts... and a scoring guide that 
identifies the thinking and writing 
requirements for that type of writing..." and 
"Essays are scored In four to six days by 
several hundred teachers at four regional 
scoring centers. A special handbook for each 
grade level provides teachers with practical 
instructional materials for each type of writing, 
including sample prompts, illustrative essays, 
and related readings.") 

3) The resulting assessment was not monitored by 
anyone with a research background in critical 
thinking. (We Infer this from the fact that model 
"strong" answers purporting to illustrate 
critical reasoning are showcased that are in 
fact patently very weak answers, containing 
virtually no reasoning at all.) 



Consider Figure 1 and Figure 2 used as illustrations of 
the nature and quality of the writing assessment pro- 
gram In an article authorized and developed by the staff 
of the California Assessment Program. It is entitled 
"Calif ornia: The State of Assessment" and was written 
for an Important national anthology. Developing Minds 
(more than 150,000 copies disseminated by ASCD). The 
show-piece article. In which these figuies occur, argues 
that the examples illustrate a "state-of-the-art teacher- 
developed writing assessment" that is sophisticated 
in "its testing, scoring, and reporting systems" and 
designed to "include only those tasks that will stimu- 
late high-quality instruction". 

There are a number of problems illustrated in these fig- 
ures that a substantive understanding of critical think- 
ing would have avoided: 

1) A description of subjective reactions was 
systematically confused with sound 
evaluative reasoning, it is important to 
distinguish questions like "Is rock music good 
music?" or "Does rock music excel as a form of 



Figure 1 

Evaluative Essay Sample 

Evaluation. Students wen asked to write an evaluative essay, make judgments about the worth of a book, television program, or type of music and 
then support their judgments with reasons and evidence. Students must consider possible criteria on which to base an evaluation, analyze their sub- 
ject in light of the criteria, and select evidence that clearly suppor i their judgments. Each student was assigned one of the following evaluative tasks: 

• To write a letter to a favorite author telling why they especially liked one of the author's books. 

• To explain why they enjoyed one television program more than any others. 

• To justify their preference for a particular type of music. 

The tasks made clear that students must argue convincingly for their preferences and not just offer unsupported opinions. 
This is a sample essay from a student who demonstrated exceptional achievement. 

Rock Around the Clock 
"Well, you're getting to the age when yen havt to Inn to be responsible!" my mother yetted out. 

"Yet, but I emit be available oil the time to Jo my appointed choree! I'm only thirteen! I wont to he with my friends, to have fun! I 
don't think that It it fair for me to baby-tit while you go run your little errmndtl" I m oppe d back. I sprinted upstairs to my room before 
my mother could start another sentence. I turned on my radio and "Shout" was playing. I noted how true the song was and I threw 
some punches at my pillow. The song ended and "Control" by Janet Jackson came on. I stopped beating my pillow. I suddenly felt at 
peace with myself. The song had slowed me down. I pondered briefly over alt the sons' **** ttahjoi me to control my feelings. The 
list was endless. So Is my devotion to rock musk and pop rock. These songs help me to express my feelings, they make me wind down, 
and above all they make me feel good. Without this music, I might have turned out to be a violent and grumpy person. 

Some of my favorite sent? are by Howard Jones, Pet Shop Boys, and Madonna. ! especially like songs that have a message in 
them, such as "Stani by Me", by Ben B. Kmg. This song tolls me to stand by the people I love and to not question them in times of 
need. Basically mis song Is telling me to believe In my friends, because they are my friends. 

My favorite type of music is rock and pop rock. Without them, mere is no way that I could survive mentally. They are with me in 
times of trouble, and best of all, they are only a step away. 

California classroom teachers wrote comments like these after reading and scoring students' evaluative essays: 

• "Evidence of clear thinking wu heavily rewarded in our scoring." 

• "I am struck by how much some students can accomplish in 45 minutes; how well they can sometimes marshal the ideas; and which how 
much flak and sparkle they can express themselves." 

• "More emphasis should be placed on critical thinking skills, supporting judgments, and tying thoughts and ideas together. Far too many 
papers digress, summarize, underdevelop. or state totally irrelevant facts." 

• "Students generally need to develop skills in giving evidence to support their judgments. I plan to spend more time on these thinking skills 
next year." 

Source: California State Department of Education 1988. 
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Figure 2 

CAP Grade 8 Direct Writing Assessment 
Achievement in Evaluation 



Percentage 
of 

California 

Grade 8 Cumulative 
Score Point Students* Percentage Description of Achievement 



0.5 



Exertional 
Achievement 



Commendable 
Achievement 



Adequate 
Achievement 



Some Evidence 
of Achievement 



Limited Evident* 
of Achievement 

1 

Minimal Evidence 
of Achievement 



No response 
Off Topic 



8.1 



25.5 



8.6 



34.1 



42.4 



19.2 



3.6 



76.5 



95.7 



99.3 



0.3 
0,5 



The student produces convincingly argued evaluation; 
identifies a subject, describes it appropriately, and asserts a 
judgment of it; gives reasons and specific evidence to support 
the argument; engages the reader immediately, moves along 
logically and coherently, and provides cloture; reflects 
awareness of reader's questions or alternative r *hiations. 

The student produces well-argued evaluation; identifies, describes, 
and judges its subject; gives reasons md evidence to support the 
argument; is engaging, logical attentive to reader 's concern; is 
more conventional or predictable than the writer of a 6. 

The student produces adequately argued evaluation; identifies 
and judges its subject; gives at least one moderately developed 
reason to support the argument; lacks the authority and polish 
of the writer of a 5 or 6; produces writing that, although 
focused and coherent, may be uneven; usually describes the 
subject more than necessary and argues a judgment less than 
necessary. 

The student states a judgment and gives one or more reasons to 
suoport it; either lists reasons without providing evidence or 
fails to argue even one reason logically or coherently. 

The student states a judgment but may describe the subject 
without evaluating it or may list irrelevant reasons or develop a 
reason in a rambling, illogical way. 

The student usually states a judgment but may describe the 
subject without stating a judgment; either gives no reasons or 
lists only one or two reasons without providing evidence; 
usually relies on weak and general personal evaluation* 



r rhii column does not total to 100% because of rounduig. 



music? 9 * (which call for objective evaluation) 
from questions like "Do you enjoy rock music?" 
or "Does rock music stir powerful emotions in 
you?* 1 (which call, not for reasoning, but for the 
description of subjective reactions). The test 
developers were apparently not clear about this 
distinction. 

2) The Assessing Teachers did not notice that 
the student felled to respond to the 
directions. The student did not develop 
evaluative reasoning, did not support his 
judgment with reasons and evidence, did not 



consider possible criteria on which to base his 
Judgment, did not analyze the subject in the 
light of the criteria, and did not select evidence 
that clearly supported his Judgment. Instead 
the student described an emotional exchange, 
asserted— without evidence— some 
questionable claims, and expressed a variety of 
subjective preferences (a fuller critique of the 
student essay is available in an appendix at the 
end of this paper). The assessing teachers were 
apparently not clear enough about the nature 
of evaluative reasoning cr the basic notions of 
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criteria, evidence, reasons, evidence, and weil- 
supported Judgment to notice the discrepancy. 

3) The California state Department of 
Education Assessment Staff did not notice 
these errors onee they were made. Instead of 
catching the errors once made, the California 
Department of Education chose to use the 
mlsgraded student essay as a showcase model 
to disseminate nationally as illustrating 
"exceptional achievement" in reasoned 
evaluation, and as a model of their assessment 
of reasoned writing. We conclude that the 
California Assessment Program is not making 
use of scholars with a background in critical 
thinking research, any of whom would surely 
have recognized the problem. 

It is essential that fundamental misconceotlons of the 
nature of critical thinking and reasoned discourse 
such as those documented above not be replicated in 
a national assessment program. Steps should be taken 
to insure that a substantive concept of critical think- 
ing and a well-supervised implementation of that con- 
cept form the basis of the finished assessment program. 



SECTION THREE 

The Four Domains of 
Critical Thinking 

What are the four component 
domains of critical thinking and the 
implications of each of these 
domains for the assessment 
of higher-order thinking? 

A. ELEMENTS OF THOUGHT. 

As soon as we move from thought which is purely asso- 
ciational and undisciplined, to thinking which is con- 
ceptual and inferential, thinking which attempts In 
some intelligible way to figure something out, to use 
the power of reason, then it is possible, and helpful, to 
think about what can be called "the elements of thought." 
The elements of thought are the basic building blocks of 
thinking, essential dimensions of reasoning whenever and 
wtterever it occurs. Working together, they shape rea- 
soning and provide a general logic to reason. We can artic- 
ulate these elements by paying close attention to what 
is implicit in the attempt on the part of the mind to fig- 
ure anything out whatsoever. Once we make them clear, 
it will be obvious that each of them can serve as an 
important touchstone or point of assessment In critical 
analysis and in the assessment of thinks g. 



Micro-skills. For each of the elements of thought there 
is a cluster of attendant basic thinking skills. Because 
they involve fundamental structures of thought, these 
skills can be characterized as micro-skills, those skills 
out of which larger-domalned critical thinking abilities 
are built. Being able to think critically about a partic- 
ular issue, then, will Include the ability to identify, 
clarify and argue for and against alternative formula- 
tions of the elements of thought. 

The basic conditions Implicit whenever we gather, con- 
ceptualize, apply, analyze, synthesize, or evaluate Infor- 
mation—the elements of thought— are as follows: 

1) Purpose, Goal, or End In View. Whenever we rea- 
son, we reason to some end, *o achieve some objective, 
to satisfy some desire or fulfill some need. One source 
of problems in reasoning is traceable to defects at the 
level of goal, purpose, or end. If the goal is unrealistic, 
for example, or contradictory to other goals we have, 
confused or muddled In some way, then the reasoning 
used to achieve it is problematic. 

An assessment of critical thinking, then, would test 
skills of being able to state an author's purpose, to iden- 
tify a plausible statement of an author's goals from a 
list provided, to raiJc formulations of an author's objec- 
tives according to which are more or less reasonable in 
light of a particular passage, to distinguish clearly 
between purposes, consequences, assumptions and 
other elements of thought. 

2) Question at Issue, or Problem to be Solved. 

Whenever we attempt to reason something out. there 
is at least one question at Issue, at least one problem 
to be solved. One area of concern for reascners, there- 
fore, will be the formulation of the question to be 
answered or problem to be solved, whether with respect 
to their own reasoning or to that of others. 

Assessing skills of mastery of this element of thought 
would test students' ability to formulate a problem in 
a clear and relevant way, to choose from among alter- 
native formulations, to discuss the merits of different 
versions of the question at issue, to recognize key com- 
mon elements In statements of different problems, to 
structure the articulation of problems so as to make 
possible lines of solution more apparent. 

3) Point of View, or Frame of Reference. Whenever 
we reason, we must reason within some point of view 
or frame of reference. Any "defect" In that raint of view 
or frame of reference is a possible source of problems 
in the reasoning. A point of view may be too narrow, too 
parochial, may be based on false or misleading analo- 
gies or metaphors, may contain contradictions, and so 
forth. 

Levels of skill here would be tested with reference to 
being able to enunciate an author's point of view In a 
passage, to adjudicate between different statements of 
that point of view, to recognize bias, narrowness, and 
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contradictions when they occur In the point of view, to 
recognize relations between the frame of reference 
being used and its implications, assumptions, and 
main concepts. 

4) The Empirical Dimension of Reasoning. Whenever 
we reason, there Is some "stuff," some phenomena 
about which we are reasoning. Any "defect," then, in the 
experiences, data, evidence, or raw material upon 
which a person's reasoning is based is a possible 
source of problems. 

Students would be tested on their ability to distin- 
guish evidence from conclusions based on that evidence, 
to give evidence themselves, to identify from a pre- 
selected list data that would support an author's posi- 
tions, data that would oppose it, data that would be 
neutral, to notice the presence or lack of relevant evi- 
dence, to recognize, to be intellectually courageous in 
recognizing (and labeling as such) mere speculation that 
goes beyond the evidence. 

5) The Conceptual Dimension of Reasoning, All rea- 
soning uses some ideas or concepts and not others. 
These concepts can Include the theories, principles, 
axioms and rules Implicit in our reasoning. Any "defect" 
in the concepts or ideas of the reasoning is a possible 
source of problems. 

The assessment of the relevant higher order thinking 
would test the ability to identify main concepts of a pas- 
sage, to choose among different versions of those con- 
cepts (some perhaps equally good), to see relations 
among concepts, to reason about the similarity of 
points of view on the basis of similarity of fundamen- 
tal concepts, to distinguish central from peripheral 
concepts, derived concepts from basic concepts, to see 
the implications of using one concept rather than 
another. 

6) Assumptions. All reasoning must begin somewhere, 
must take some things for granted. Any "defect" in 
the assumptions or presuppositions with which the rea- 
sonliig begins Is a possible source of problems. 

Assessing skills of reasoning about assumptions would 
test the ability to identify assumptions underlying 
given inferences, points of view, and goals, to evaluate 
the accuracy of different formulations of the assump- 
tions, to distinguish between assumptions and infer- 
ences, to rank assumptions with respect to their plau- 
sibility, to be intellectually fairmlnded by choosing 
the most plausible version of assumptions underlying 
points of view with which they disagree. 

7) implications and Consequences. No matter where 
we stop our reasoning, it will always have further 
implications and consequences. As reasoning develops, 
statements will logically be entailed by it. Any "defect" 
in the implications or consequences of our reasoning is 
a possible source of problems. 



Skills to be assessed would Include the ability to iden- 
tify important implications, to do so by selecting from 
a list of possible Implications, tc make fine discrimi- 
nations among necessary, probable, and improbable 
consequences, to distinguish between implications and 
assumptions, to recognize the weakness of an author's 
position as shown by the implauslblllty of its implica- 
tions, to exercise intellectual falrmlndedness In dis- 
criminating between the likelihood of dire and mild 
consequences of an action to which one is opposed. 

8) Inferences. Reasoning proceeds by steps in which 
we reason as follows: "Because this is so, that also is 
so (or probably so)," or "Since this, therefore that "Any 
"defect" In such inferences is a possible problem in our 
reasoning. 

Assessment would test students' ability to recognize 
faulty and Justified inferences in a passage, to rank 
inferences with respect to both their plausibility and 
their relevance, to make good inferences in their own 
reasoning, to discriminate among various formula- 
tions of an author's inferences with respect to which is 
most accurate, to take something they do not believe but 
to entertain it for the sake of argument and draw rea- 
sonable Inferences from it. 

Assessment of Elements of Thought. Any program for 
the assessment of critical thinking skills must itself be 
assessed In terms of its validity and reliability in test- 
ing for the ability to think about, and in terms of, the 
elements of thought. These abilities can be successfully 
assessed In three related ways: by a restricted use of 
standard multiple-choice items, by multiple-rating 
items, and by short essay items. Both multiple-choice 
and multiple -rating items are machlne-gradable, while 
essay items are not. 

Although our recommendations about the content of the 
assessment will be spelled out in detail in Section 
Four, some of these can be anticipated here with 
respect to the assessment of reasoning abilities cen- 
tering around the elements of thought. 

Multiple choice testing (as In the existing Watson- 
Glaser(>iticolThtnktngAppraisaloT the Cornell Critical 
Thinking Tests) Is an important part of an assessment 
of critical thinking, but its legitimate use is restricted 
to testing only the most basic skills of identifying and 
recognizing elements of thought, and then only as they 
occur in relatively short and unambiguous excerpts. 

Within this domain, multiple-choice questions will 
require students: 

• tc identify an author's purp03v. in a passage; 

• to rate selected Inferences as Justified, probably 
true, insufficiently evidenced, probably false, 
unjustified: 

• to select among formulations of the problem at 
issue In a passage those that are clearly 
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reasonable, probably reasonable, probably 
unreasonable, clearly unreasonable; 

to recognize unstated assumptions; 

• to distinguish evidence from hypotheses and 
conclusions; 

• to rate described evidence as reliable, probably 
reliable, probably not reliable, unreliable. 

B. MACRO-ABILrnES. 

The elements of thought do not exist in isolation from 
one another, nor— more importantly for the concept of 
an assessment procedure— do they exist outside a par- 
ticular context of application. In the practice of good crit- 
ical thinking, skills more closely associated with ele- 
ments of thought are orchestrated into larger-domained 
abilities, called macro-abilities, which are applied to 
thinking about complex and sometimes ambiguous 
issues, problems, decisions, theories, states of affairs, 
social institutions, and human artifacts. 

These critical thinking macro-abilities include being 
skillful at: 

(1) refining generalizations and avoiding over- 
simplifications, 

(2) comparing analogous situations: transferring 
insights into new contexts, 

(3) developing one's perspective: creating or 
exploring the implications of beliefs, 
arguments, or theories, 

(4) clarifying issues, conclusions, or beliefs, 

(5) clarifying and analyzing the meanings of words 
and phrases, (constructing and clarifying 
interpretations] 

(6) developing criteria for evaluation: clarifying 
values and standards, 

(7) evaluating the credibility of sources of 
information, 

(8) questioning deeply: raising and pursuing root 
or significant questions, 

(9) analyzing or evaluating arguments, 
interpretations, beliefs, or theories, 

(10) generating or assessing solutions, 

(11) analyzing or evaluating actions or policies, 

(12) reasoning analogically: comparing perspectives, 
interpretations, or theories, 

(13) reasoning dialectically: evaluating perspectives, 
interpretations, or theories, 

(14) reading critically: constructing an accurate 
interpretation of, understanding the elements 
of thought in, and evaluating, the reasoning of 
a text. 



(15) listening critically: constructing an accurate 
interpretation of, understanding the elements 
of thought in, and evaluating,the reasoning of 
an oral communication, 

(16) writing critically: creating, developing, 
clarifying and conveying, in written form, 
the logic of one's thinking, 

(17) speaking critically: creating, developing, 
clarifying and conveying, in spoken form, the 
logic of one's thinking. 

Macro-abilities like these play a central role in a rich and 
substantive concept of critical thinking. They are essen- 
tial to approaching actual issues, problems and situa- 
tions in a rational way. Understanding the rights and 
duties of citizenship, for example, requires that one at 
least have the ability to compare perspectives and inter- 
pretations, to read and listen critically, to analyze and 
evaluate policies. In fact there is no macro-abUtty on the 
list that would not be relevant or even crucial to think- 
ing deeply about the rights and duties of citizenship. 
Similarly, the capacity to make sound decisions, to 
participate knowledgeably in the workplace, to function 
as part of a global economy, to master the content in 
anything as complex as the academic disciplines, to 
apply those disciplines to real-life situations, to make 
insightful cross-disciplinary connections, to commu- 
nicate effectively— each of these relies in a fundamen- 
tal way on having a significant number of the macro-abil- 
ities listed. Take, for example, the capacity to make 
sound decisions: such decision-making is hardly pos- 
sible without an attendant ability to (going down the list 
of macro-abilities in order) refine generalizations, com- 
pare analogous situations, develop one's perspective, 
clarify Issues, and so forth. 

The last four macro-abilities listed— the ability to read, 
write, listen, and speak, each In a critical, informed, 
constructive way. at a post-secondary level of sophis- 
tication— are best considered not as in the usual model, 
not as manifestations of thinking already accomplished, 
but as being themselves actual modes of constructive 
thinking. As such, they are structured amalgams of ele- 
mentary skills together with any number of other 
macro-abilities. 

Assessment of macro-abilities is essential to assess- 
ment of critical thinking. Since these are the abilities 
implicit in the realistic use of thinking, no assessment 
tool that fails to assess a significant number of these 
abilities could justifiably be called an assessment of 
higher-order thinking. The assessment, moreover, 
needs to address such abilities directly (rather than 
through secondary indicators), systematically (rather 
than haphazardly as a result of an attempt to assess 
other variables like academic achievement), and in 
settings as authentic as possible given the requirement 
of uniform, relevant grading. 
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Assessment of macro-abilities that meets these four 
criteria cannot be accomplished within the confines of 
a standard multiple-choice-type test. It can be accom- 
plished, however, for all of the macro-abilities (except 
those having to do with oral communication), by means 
of a combination of machine-gradable multiple-rating 
items and essay items. 

For airy macro-ability, there will be dimensions of the 
ability that are generative and other dimensions of it 
that are selective. In trying to solve a real problem, for 
example, a good deal clone's thinking Is devoted to gen- 
erating a formulation of the problem that will make it 
more susceptible to solution. Another, and quite dif- 
ferent, aspect of problem solving, Is the ability to select, 
from among a large variety of possibilities, that avenue 
of thought which will most likely result in a solution 
Students who are trained using a rich, substantive 
concept of critical thinking tend to Improve in both 
dimensions of this ability, and both are genuine dimen- 
sions of real problem-solving. 

The selective dimensions of an ability can be assessed 
accurately, even in complex, ambiguous, and subtle 
cases, using multiple-rating items. The generative 
dimension, on the other hand, cannot. Since it requires 
students to come up with their own critical thinking 
approaches within that macro-ability, this dimension 
can be assessed adequately ovily by carefully con- 
su acted and carefully graded essay tests. Details of the 
assessment and samples of assessment items will be 
presentee, to Section Four. 

C. AFFECTIVE DIMENSIONS. 
Higher order thinking requires more than higher order 
thinking skills. Critical thinking, to any substantive 
sense, includes more than macro-abilities. The concept 
also includes, to a crucial way, certain attitudes, dis- 
positions, passions, traits of mind. These affective 
dimensions are not merely important to critical think- 
ing, they are essential to the effective use of higher order 
thinking to real settings. 

These affective dimensions Include: 

(1) thinking independently, 

(2) exercising falrmindedness, 

(3) developing Insight into egocentriclty and 
sociocentriclty, 

(4) developing intellectual humility and 
suspending Judgment, 

(5) developing Intellectual courage, 

(6) developing intellectual good faith and integrity, 

(7) developing intellectual perseverance, 

(8) developing confidence to reason, 



(9) exploring thoughts underlying feelings and 
feelings underlying thoughts, 

(10) developing intellectual curiosity. 

Without intellectual perseverance, one could not 
solve the complicated, multi-faceted problems one con- 
fronts to industry. Without intellectual courage, one 
could not maintain a defense of citizenship rights to the 
face of scare tactics. Without falrmindedness, one 
could not enter into another's point of view and thus 
would lack that empathetic understanding necessary 
for a reasonable approach to living in a pluralistic 
society. Without developing Insight Into egocentric- 
lty and sociocentriclty one could employ one's rea- 
soning skills to a merely self-serving and prejudiced 
way. Without confidence In reason one could not 
adequately address those complex and frequently 
ambiguous real-life problems that require reasonable 
decisions to the face of crucial uncertainties. 

Assessment of affective dimensions of critical think- 
ing is an important part of an assessment of higher- 
order thinking. An initial problem Is that from the fact 
that all thftse dimensions are essential, it does not fol- 
low that all are directly testable, nor does it follow that 
any of them is easily testable. For some of these affec- 
tive dimensions (intellectual perseverance, for example), 
airy testing would have to take place over an appro- 
priately long period of time and thus could not be 
legitimately assessed at all during a time-frame suitable 
for a national test. 

Nevertheless, a number of affective dimensions can be 
assessed to a relatively straightforward way using essay 
items and. especially, machine-gradable multiple-rating 
items. 

"Reasoning Within Conflicting Points of View," a central 
aspect of the disposition of falrmindedness, is already 
being assessed on the revised version of the Watson- 
Closer Critical Thinking Appraisal This section of the 
Appraisal asks students to select the strongest (Le.. the 
most defensible) argument to favor of each side of a pair 
of conflicting and sometimes emotionally charged points 
of view. Proficiency on these items indicates a falrmtod- 
ed willingness to distinguish the concept of reason- 
able defensibmty from that of personal beOtf. 

Multiple-rating items are currently being prepared that 
address aspects of Intellectual courage, other aspects 
of falrmindedness. aspects of intellectual humility, 
and aspects of the development of insight into one's own 
egocentricity and sociocentriclty. 

D. IIV1EIIECTVAL STANDARDS. 
In any domain where assessment is taking place, there 
are standards that are implicit to the assessment. 
Higher order thinking is thinking that meets universal 
Intellectual standards. Thus, when assessing a student's 
ability to compare and evaluate perspectives (a rnacro- 
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ability) and to do so with fairmindedness (a trait of 
mind), we would Judge whether she had made such 
evaluation In a relevant and consistent way, with 
attenUon to accuracy, fairness, and completeness In 
describing each perspective, and with a sensitivity to the 
degree of precision appropriate to the topic. We would 
assess critical thinking about and in terms of the ele- 
ments of thought in very much the same way: to Judge 
a person's skill at recognizing the frame of reference 
underlying an Issue, we would want to Judge whether 
she could see relevant alternatives, whether the frame 
of reference she identified fits the available evidence, 
whether her answer was deep or merely mechanical, 
clear or vague, biased or fair. Intellectual standards 
apply to thinking In every subject. 

The process of learning to teach so as to foster critical 
thinking is the very process by means of which one 
establishes intellectual standards for assessing think- 
ing, and, by extension, for assessing instruction itself. 

Such standards are more useful if they are made 
explicit— to the students who are taking the test, to 
those doing the assessing, and to classroom teachers. 
Making standards explicit benefits student test-takers 
because they can then see that there are standards, that 
the standards are not arbitrary ones, and that under- 



standing the standards gives them an Insight into 
what good critical thinking is. It benefits those doing the 
assessing because, In addition to the reasons already 
mentioned, it fosters both a uniformity In grading and 
a strong correlation between the grade and the skills 
being graded. Judging a response by how clearly and 
completely it states a position, for example, Is using a 
critical-thinking Standard and dictates a certain level 
of assessment; Judging a response by how concisely or 
how elegantly it states a position, on the other hand, is 
using a standard that is inappropriate to critical think- 
ing assessment. Explicit standards— part of a rich and 
substantive concept of critical thinking— might have 
avoided at least some the mistaken assessment on the 
California Assessment Program, cited earlier (see p. 9). 
Thus, making standards explicit promotes both the 
reliability and the validity of the assessment-vehicle. 
Finally, it benefits classroom teachers because such 
standards can readily be built Into classroom Instruc- 
tion. The standards, after all, are those Implicit in 
teaching for higher order thinking skills; they are there- 
fore Invaluable both for teachers to use explicitly with 
their classes and— an essential feature of critical-think- 
ing-internalized— for students to learn to use as part of 
assessing themselves. 



Intellectual Standards 
That Apply to Thinking in Every Subject 



Thinking that is: Thinking that is: 

Clear vs Unclear 

Precise vs Imprecise 

Specific vs Vague 

Accurate vs Inaccurate 

Relevant vs Irrelevant 

Plausible vs Implausible 

Consistent vs Inconsistent 

Logical vs Illogical 

Deep vs Superficial 

Broad vs Narrow 

Complete vs Incomplete 

Significant vs Trivial 

Adequate (for purpose) vs Inadequate 

Fair vs Biased or One-Sided 
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SECTION FOUR 

RECOMMENDATIONS OF THE 
CENTER FOR CRITICAL 
THINKING 

What is the simplest solution to the 
design qf a process to assess higher- 
order thinking at the post-secondary 

level? 

In this section we will (A) briefly survey existing assess- 
ment tools; (B) make recommendations regarding the 
substance and format of a national assessm e nt tool- 
including the critical thinking domains to be assessed, 
the varieties of assessment strategies to be used (togeth- 
er with sample test items), and the dual interdisci- 
plinary and intxadlscipllnaiy scope of the assessment— 
(C) appraise the value of the proposed assessment 
strategy for the reform of instruction, and (D) make rec- 
ommendations regarding the implementation of the 
assessment. 

A. Existing Assessment Tools. 

There are limitations In all twelve of the commercially 
available critical thinking tests as instruments for 
assessing higher order thinking: 

Cornell Class Reasoning Test, Form X (1964) 

Cornell Conditional Reasoning Test, Form X 
(1964) 

Cornell Critical Thinking Test, Level X ( 1985) 

Cornell Critical Thinking Test. Level Z (1985) 

The Ennis-Welr Critical Thinking Essay Test 
(1985) 

Judgement: Deductive Logic and Assumption 
Recognition (1971) 

Logical Reasoning (1955) 

New Jersey Test of Reasoning Skills (1983) 

Ross Test of Higher Cognitive Processes (1976) 

Test on Appraising Observations (1983) 

Test of Enquiry Skills ( 1979) 

Watson-Glaser Critical Thinking-Appraisal 
(1980) 

In addition there are limitations in all of the other 
available "higher studies" tests which might be taken 
as a possible model for the assessing of higher order 
thinking: the SAT. LSAT, the Test of Academic Aptitude 
(British), the Graduate Record Exam, the Com- 
monwealth Secondary Scholarships Exam (Australia). 
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We do not have the space here to review each of these 
tests one-by-one. Instead we will summarize the gen- 
eral situation as we see it. 

Though aspects and dimensions of critical thinking 
are tested, some more and some less. In a'J of the 
above tests, none has been designed with the 21 criteria 
above (p. 3) In mind. Most importantly, none was 
designed to serve as a national assessment tool which 
establishes national standards In higher order think- 
ing and in motivating and guiding instruction so as to 
lead to the achievement of the goal: The proportion of 
college graduates who demonstrate an advanced abil- 
ity to think critically, communicate effectively, and 
solve problems will increase substantially." 

Behind none of these tests was there a comprehensive 
model for the elements of thought, the macro-abilities 
of critical thinking, or the affective dispositions (as we 
have here provided). The relative recent ness of the 
bulk of scholarship in critical thinking makes it unlike- 
ly that long-established tests will fill the bill. 

Of course any new test for assessing higher order 
thinking should be based on a thorough review of 
established test strategies to incorporate those with sig- 
nificant application. 

Given the need for assessment on the basis of a rich and 
substantive concept of critical thinking, there are two 
areas where competing values and objectives come 
into play. 

Th«i first concerns the substance and format of the test 
itself: Which domains exactly are to be covered, and with 
whi.t emphases? What kinds of question will be asked? 
Will it be interdisciplinary or intradlsciplinary? What 
kivid of assessment question best relate to testing for 
s dlls of citizenship and the challenges of the workplace? 

The second area concerns the Implementation of the 

test and how it Is conceived: Should it be value-added 
or simply criterion-referenced? Who will do the assess- 
ing and who will be assessed? How much will the 
assessment cost and who will pay for it? How often will 
the test be given? 

Some of these are difficult questions, with genuine 
values and goals on different sides, where reasonable 
cases can be made for more than one position. Others 
of these questions are clearer, especially once the objec- 
tives of the test as a whole are brought into focus. 

B. Substance and Format. 

The overall recommendations of the Center For Critical 
thinking are set forward below. 

(1) DOMAINS TO BE ASSESSED. 
The national assessment of higher order thinking at the 
post-secondary level must test for a rich and sub- 
stantive concept of critical thinking, and this testing 
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: iust be geared to assessment within all four domains 
of critical thinking. 

(q) Element! of thought 
Skills of identifying, explicating, and using the ele- 
ments of thought need to be assessed. They are 
necessary for any of the macro-abilities to be 
employed with precision, depth or accuracy. They are 
required if essential affective traits are to be rooted 
in solid, locatable, Intellectual skills and the concepts 
they presuppose. 

Lack of a solid grounding In these skills, and the 
concepts behind these skills, results In thinking 
which, good Intentions notwithstanding, is far 
removed from the close, careful reasoning demand- 
ed by the rigors of higher order thinking. Even 
among testing personnel, lack of the Informed use 
of these concepts is part of what results to such 
poor assessment-tools and -grading as we found to 
the California Direct Writing Assessment. 

Critical thinking to students requires them to be 
able to perform well on items testing a list of skills 
that center around the elements of thought: 

• identify a plausible statement of a writer's 
purpose; 

• rank formulations of an author's objectives; 

• distinguish clearly between purposes, 
consequences, assumptions, and 
inferences; 

• choose the most reasonable statement of 
the problem an author Is addressing; 

• discuss reasonably the merits of different 
versions of the question at issue; 

• recognize key common elements to 
formulations of different problems; 

• give a clear articulation of an author's point 
of view; 

• decide the most reasonable statement of an 
author's point of view; 

• recognize bias, narrowness, and 
contradictions to the point of view behind 
an excerpt; 

• identify assumptions and Implications of a 
writer's point of view; 

• distinguish evidence from conclusions 
based on that evidence; 

• give evidence to back up their position to an 
essay; 

• recognize data that would support, data 
that would oppose, and data that would be 
neutral with respect to, an author's 
position; 

• recognize conclusions that go beyond the 
evidence; 



note, to an evaluative essay, the absence of 

evidence to an excerpt; 

identify the main concepts to a passage; 

distinguish central from peripheral 
concepts; 

identify the assumption underlying a given 
Inference; 

evaluate the aptness of different versions of 
an assumption; 

choose the most reasonable statement of a 
background theory involved to a passage; 
distinguish between Inferences and 
assumptions; 

rank different formulations of assumptions 
with respect to which is the most 
reasonable; 

identify crucial Implications of a passage; 

discriminate between consequences that 
are necessary, probable, and Improbable; 

evaluate an author's Inferences; 

make, to an evaluative essay, Justified 
inferences; 

choose the most accurate version of an 
author's inferences; 

draw reasonable Inferences from positions 
they disagree with. 



(b) Macro-abilities 
Macro-abilities, grounded to a thorough familiar- 
ity with the elements of thought, are the activities 
we actually use to perform our higher order think- 
ing. Abilities like clarifying values and standards, 
comparing analogous situations, generating and 
assessing solutions, analyzing and evaluating 
actions or policies are the stufT of reasoning. They 
are the means whereby decisions are to be made, 
problems are to be solved, Industry is to be 
strengthened, and understanding of rights and 
responsibilities deepened. 

The macro-abilities of critical reading and critical 
writing are keystones of any process to assess 
higher order thinking to that each of them, when 
considered at the post-secondary level is permeated 
by other critical thinking macro-abilities. It is not 
as if we read and clarify values, read and compare 
analogous situations, write and generate solutions. 
To read critically is to clarify values, compare anal- 
ogous situations, and to exercise the other macro- 
abilities as well; to write is to generate solutions and 
much more besides. 

Assessment of proficiency in the macro-abilities 
can be keyed to student performance on test items 
geared to as many of the macro-abilities listed on 
p. 14 as Is feasible given the time constraints of the 
test. 
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(c) Affective trait* 

Without assessing affective traits, only a dimin- 
ished idea of critical thinking will be addressed. 

What allows us to confront our prejudices and ana- 
lytically break them down is not Just macro-abilities 
but a commitment to use the macro-abilities in this 
regard. What allows us to solve our problems in a suf- 
ficiently diligent way as to address complicated and 
intricate real-life problems, is again not Just cogni- 
tive abilities. It is intellectual perseverance— a drive, 
a disposition, an affective trait. A similar point can 
be made for each of the intellectual traits which are 
the driving force behind sound and penetrating 
reasoning. 

Assessment of the affective dimensions will con- 
centrate on those aspects it is plausible to test for 
within the constraints imposed by a national assess- 
ment. These will include aspects of fair-mindedness, 
of the willingness to suspend Judgment, of intel- 
lectual courage and intellectual integrity. 

(d) Intellectual Standards, 
Assessment has to involve explicit universal stan- 
dards. If we are not testing students' abilities to be 
relevant, precise, logical, consistent, and the rest, 
then we are not assessing students' abilities to 
engage in higher order thinking. 

And if testing personnel do not employ these same 
explicit standards, then they are grading for some- 
thing other than higher order thinking. 

Relative mastery of these intellectual standards 
requires students to be able to 

• recognize clarity vs. unclarity; 

• distinguish accurate from inaccurate 
accounts; 

• decide when a statement is relevant or 
irrelevant to a given point; 

• identify inconsistent positions as well as 
(relatively) consistent ones; 

• discriminate deep, complete, and 
significant accounts from those that 
are superficial, fragmentary, and trivial; 

• evaluate responses with respect to their 
fairness; 

• prefer well-evidenced accounts to 
accounts that are unsupported by 
evidence; 

• tell good reasons from bad. 

(2) VARIETIES OF ASSESSMENT STRATEGIES. 
The assessment should contain items of three varieties: 
(a) machine-gradable multiple choice items; (b)machine- 
gradable multiple-rating items; (c) essay items. 



(a) Multiple-choice items. 

Legitimate use of multiple-choice items on the 
assessment is limited. This type of item Is geared 
toward relatively straightforward skills of reasoning, 
particularly with respect to recognizing elements of 
thought, distinguishing one element of thought 
from another, and recognizing clear examples of 
faulty reasoning. 

Two detailed samples of assessment items follow 
(the first, figure 3, is on Inferences, the second, fig- 
ure 4, on Recognition of Assumptions: 

Other abbreviated samples of appropriate multiple- 
choice items are as follows: 

(I) In the following excerpt, mark E for 
each iterr* that is a piece of empirical 
evidence; mark C for each item that is 
a conclusion based on evidence; mark 
N for each item that is neither.... 

(II) In this test, each exercise consists of 
several statements (premises) followed 
by several suggested conclusions... If 
you think the conclusion necessarily 
follows from the statements given, make 
a heavy black mark under 
"CONCLUSION FOLLOWS"; if you think 
it is not a necessary conclusion, put a 
mark under "CONCLUSION DOES NOT 
FOLLOW." 

(ill) The following is a list of possible 
findings in relation to the experiment 
quoted above. For each, say whether it 
would support the author's hypothesis, 
oppose the author's hypothesis, or be 
neutral with respect to the author's 
hypothesis... 

(iv) Below is a series of questions. Each 
question Is followed by several reasons. 
For the purpose of this test, you are to 
regard each reason as true. The 
problem then is to decide whether it is a 
strong reason or a weak reason... 

(v) Which of the following conclusions is C 
completely supported by the stated 
evidence, P partially supported by the 
stated evidence, or U unsupported by 
the stated evidence? 

(vl) Which of the following is an 

implication of the author's position 
in the passage cited? 



19 



Figure 3 

Inference 

Directions: An Inference Is a conclusion a person can draw from certain observed or supposed facta. For exam- 
ple, If the lights are on In a house and music can be heard coming from the house, a person might Infer that 
someone Is at home. But this inference may or may not be correct Possibly the people In the house did not turn 
off the lights and the radio when they left the house. 

In this test, each exercise beg tns with a statement of facts that you aw to regard as true. After each statement 
of facts you will find several possible inferencea-that Is, conclusions that some persons might draw from the 
stated facts. Examine each inference separately and make a decision as to its degree of truth or falsity. 

For each Inference you will find spaces on the answer sheet labeled J, PJ, ID, PU, and U. For each Inference 
make a mark on the answer sheet under the appropriate heading as follows: 

J If you think the Inference is definitely JUSTIFIED: that It properly follows beyond a reasonable doubt 
from the statement of facts given. 

PJ if you think the Inference is PROBABLY JUSTIFIED; that It Is more likely to be true than false in the 
light of the facts given. 

ID if you decide that there are INSUFFICIENT DATA; that you cannot tell from the facts given whether the 
inference is Justified or not; if the facts provide no basis forjudging one way or the other. 

PU if you think the Inference is PROEABLY UNJUSTIFIED; that it is more likely to be false than true In 
the light of the facts given. 

U if you think the inference is definitely UNJUSTIFIED; that it does not follow, either because It 
misinterprets the facts given, or because it contradicts the facts or necessary Inferences from those 
facts. 

Example 

The first newspaper in America edited by Ben Hani *, appeared in Boston on September 25, 1 690, and was banned 
the same day by Governor Simon Bradstreet The editor's subsequent long fight to continue to publish his paper 
and print what he wished marks an Important episode In the continuing struggle to maintain a free press. 

1) The editor of the first American newspaper died within a few days after his paper was banned on September 
25,1690. 

2) Information about the first Issue of Ben Harris's newspaper promptly came to Governor Bradstreet's atten- 
tion. 

3) The editor of this paper wrote articles criticizing Governor Bradstreet 

4) Ben Harris persisted in holding to some of his alms. 

5) Governor Bradstreet objected to some of the items published in Ben Harris's paper. 
In the above example: 

Inference 1 Is (U) unjustified because in the facts given it mentions "the editor's long fight to continue to pub- 
lish his paper..." 

Inference 2 is (J) Justified because the facts state that the first newspaper appeared on September 25, 1690, and was 
banned the same day by the Governor 

Regarding Inference 3, there is no information given about the precise nature of the articles appearing in the paper, 
thus (ID) Insufficient data 

Regarding inference 4, the facts given mention "the editor's subsequent long fight to continue to publish his news- 
paper and print what he wished..."; thus (J) Justified. 

Inference 5 is deemed (PJ) probably Justified because the Governor banned the paper the day it appeared. However 
this is PJ rather that J because there may have been reasons for the ban other than objections to some of the items 
that appeared In the paper* 
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Figure 4 

Recognition of Assumptions 

Directions 

Careful reasonera often find it necessary to complete partially stated arguments In order to evaluate those 
arguments. For example, someone might say, "John la selfish: we are good friends, but he never lends 
me money." The conclusion that "John la selfish" Is supported by two explicit claims: 

') John never lends me money. 

2) John and I are good friends. 

But an Important part of the argument was left out: 

3) People who never lend money to their good friends are selfish. 

This third assertion Is an unstated assumption of the argument. 

In this test each exercise begins with a brief argument Each argument is followed by three numbered 
statements. Examine each of the numbered statements Individually and make a decision about it's log- 
ical relationship to the argument For each numbered statement there are spaces on your answer sheet 
labeled: EC, UA. and N. Select Just one of the following alternatives for each numbered statement, and 
make a mark on your answer sheet under the appropriate heading: 

EC If you think the idea expressed In the numbered statement Is an explicit claim made in the 
argument (even if the wording Is not the same). 

UA If you think the Idea expressed in the numbered statement is a probable unstated assumption 
of the argument. 

N If you think the Idea expressed in the numbered statement Is .wither an explicit claim nor an 
unstated assumption of the argument 

Example: 

Argument: "We need to save time In getting there, so we'd better go by plane." 

1) Going by plane will take less time than going by some other means of transportation. 

[Saving time Is given as a reason for going by plane; this only makes sense If the person giving the argu- 
ment believes that going by plane would take less time than other available means of transportation. So 
the idea expressed here Is an unstated assumption of the quoted argument] (UA) 

2) We should try to cut down how long we spend travelling to our destination. 

(The Idea expressed here Is directly asserted, though In different words, In the argument, so It Is not an 
unstated assumptions of the argument; ra ner, It is an explicit claim made in the argument.) (EC) 

3) Travel by plane is more convenient than travel by train. 

[No mention Is made in the argument of either trains or convenience. The idea expressed here is neither 
an explicit claim nor an unstated assumption of the argument] (N) 



(b) Multiple-Rating items. 
Though the use of multiple-choice questions is 
justified In assessing mlcrosklUs, the bulk of the 
machine gradable items will be multiple-rating rather 
than multiple choice. 

Multiple-rating items require students to evaluate 
each item rather than to select a single correct 
answer. They thus gauge abilities at the highest level 
of Bloom's Taxonomy rather than those at the bot- 
tom. Multiple-rating items allow one to ask ques- 
tions where any number of answers from a provided 
list may be correct, or Incorrect. It further allows 
students to rank; from a number of possibilities pro- 
vided, those that are more correct. For example, 
teachers of critical thinking commonly grade — A, 



B. C. D, or F — the overall reasoning ability dis- 
played In a series of student writing samples. This 
is in effect a multiple-rating assessment: the t ach- 
er takes each writing sample and rates it. with no 
pre-deterntinvd guidelines about how many will 
be A's. B's. etc. It Is perfectly possible, on any given 
sample, that all items will be rated high, medium, 
or low. Thus, students can be assessed on their abil- 
ity to grade — again, A, B, C, D, F — passages with 
respect to any dimension of critical thinking dis- 
played to the passage. 

The same list of possible answers can pertain to any 
number of Independent test Items. Thus, a list of 
twenty possibilities can be provided, and students can 
be asked to choose the appropriate response from that 
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list to six different questions. There Is no restriction 
on the number of times a given answer may be cor- 
rect Nor Is there any guarantee that there will be a 
reasonable answer on the list to every question. This 
allows much more subtle testing and grading. 
Moreover, guessing, using the process of elimina- 
tion, and scoring well because of test-taking skills are 
all but impossible. 

By including clearly unreasonable choices among 
the multiple-rating possibilities, a grade can be 
much more sensitive to the degree of a macro-abil- 
ity or to the intensity of an affective dimension. 
Thus, if there are five possible answers to a given 
question, they need not be graded 5, 4, 3, 2, 1. 
Rather, they may be graded, say, 5, 4, 1, 1, -3. 

We have provided two detailed samples of multiple- 
rating items. Figure 5 Is on Reasoning Within Con- 
flicting Points of View (and thus Is an assessment 
of an aspect of the affective trait of falrmlnded- 
ness) and Figure 6 Is on Comparing Analogous 
Situations (and is thus an assessment of a macro- 
ability). Each sample is limited here by having only 
four possible answers, a limitation that would not 
obtain on an actual test. 

The following is a list of abbreviated samples of multi- 
ple-rating items, having to do with elements of thought, 
with macro-abilities, with affective dimensions, and 
with Intellectual standards. 

Multiple-Rating Items, Elements of Thought 
(1) Here is a list of formulations of the writer's 
objectives in this excerpt. Rank them from 1 to 5 



with respect to which is the most reasonable in the 
light of the quoted passage... 

(11) For each of the underlined passages in the 
excerpts below, mark P on the answer sheet if it is 
a statement of the writer's PURPOSE. C If it Is a 
statement of the CONSEQUENCES. A if it Is a state- 
ment of the writer's ASSUMPTIONS, and I if it is an 
INFERENCE the writer is making. 

(ill) Which of the following would the author most 
likely give as the statement of the problem she is 
attempting to solve? (lv) Read the excerpt: from 
the following list, identify the most plausible state- 
ment of the writer's purpose... 

(▼) Of the following statements of the author's point 
of view In this passage, select the one from the fol- 
lowing list that is both most reasonable and most rel- 
evant to the passage.... 

(▼D List A below Is a list of various possible state- 
ments of the writer's point of view in the quoted pas- 
sage; List B Is a list that Includes possible assump- 
tions and Implications of those points of view. 
Match the items on list A with the items on list B. .. 

(vii) v/hich of the following are main concepts In the 
passage cited; which are peripheral concepts? 

(▼ill) For each inference below, decide whether the 
accompanying statement is U an unstated assump- 
tion, A an assertion, or N neither... 

(tx) Rank the following items on a scale of 1 to 5 
according to how reasonable it Is as a statement of 
the author's assumptions... 



Figure 8: Reasoning Within Conflicting Points of View 

Directions: In the tollowing questions, rank the answers in order of reasonability. In each case you are being tsked to rank 
answers as to which is the strongest argument in favor of a position. By the strongest we menn the one that is most 
defensible, not necessarily the one which claims the most To rank a defense for a position high does not mean that you 
actually hold that position but only that if you had to defend it before an audience of unbiased and openminded people, the 
options you rank higher would be easier to defend on rational grounds than the ones you rait't lower. 

»•» Children under the age of twelve should have all of their Important decisions made for them by their parents and 
other appropriate adults because: 

1 ) allowing them to make all important decisions for themselves will encourage false pride and stubbornness. 

2) allowing them to make all important decisions for themselves will undermine parental respect and authority 

3) children are not mature enough to make all important decisions for themselves 

4) children should not be expected to take life's problems so seriously until they grow up 

5) children can be expected to make grave mistakes, some of which could harm them for life 

52) Children under the age of twelve should make some Important decisions for tbemselvea because: 

1) children are less prejudiced than adults and more open to the truth 

2) children spend a lot of time watching T.V. so they know a lot about what is going on in the world 

3) children are likely to make many reasonable decisions affecting themselves 

4) children will become depressed if they are not allowed to make some important decisions 

5) children will be more apt to become responsible adults if they are allowed to make some important decisions for 
themselves as they are growing up 
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Figure 6: Compiling Analogous Situations 

"Having a population to study instead of an individual fossil is enormously important. No two people today sre exactly alike; 
no two Austratopithecines were either. It is for that reason that drawing conclusions from a single fossil is risky. Measurements 
taken cf it, and theories spun off as t result of those measurements, may be misleading because the psrt being measured may 
not be typical It is only when a large number of specimens is available that all their variations can be taken into account, and 
a norm derived from them. If a visitor from outer space were to desctibe and name Homo sapiens sapiens by examining one 
skeleton, that of a short, squat, heavy-boned New Guinea tribesman, he would certainly be excused if he set up another species 
on the basis of a second skeleton discovered later a few thousand miles away— that of a seven-foot, slender-boned Watutsi 
tribesman from central Africa" (Edey, The Emergence of Man, pp. 47-48). 

The author of the above passage makes an analogy between an anthropologist studying fossils arJ a visitor from outer space 
studying one or two single skeletons. Rank each of the following comments 1 to 3, according to whether it would be crucial 
to judging the strength of the analogy for die point die author is making. Give a comment a 3 if it is CRUCIAL in judging the 
worth of the analogy, give it a 1 if it is IRRELEVANT to judging the worth of the analogy; give it a 2 if it lies in between. 

(s) The analogy illustrates the point well because in both cases we are called upon to draw general conclusions based on 
a limited sample. The more items you have in your sample, the more justified your generalization will be. 

(b) It is a bad analogy because the visitors from outer space would draw the same erroneous conclusion even if they had 
a whole population of New Guinea tribesmen to study. 

(c) It is a good analogy but it shows that we need not simply more fossils of Australopithecus, but fossils of it from 
'ther geographical areas. 

(d) It is a bad analogy because we have no idea what visitors from outer space would conclude from seeing a skeleton of 
a New Guinea tribesman. The visitors might refrain from making the generalization for the same reason that makes 
the author say it is "risky." 



(i) Look at each of the statements below as a pos- 
sible consequence of the writer's position in the 
excerpt cited. Rank each statement on a scale of 1 
to 7, where 7 means that you consider the state- 
ment a necessary consequence of the passage, and 
1 means that you consider the statement a highly 
unlikely consequence of the passage. 

(si) Each of the following is an inference one might 
draw from the passage. Rank each one on a scale 
from 1 to 5. according to whether it is completely 
Justified (5) or completely unjustified (1)... 

(xtl) Which of the following is the most accurate for- 
mulation of the author's Inference in the cited passage? 

Multiple-Rating Items, Macro-Abilities. 
(xill) Which of the following would be relevant to 
deciding whether A Is a credible source of infor- 
mation on the topic...? 

Gdv) Here is a list of observations about the behav- 
ior of X*s, made by a .responsible Investigator. Which 
of the items from the following list would be a Jus- 
tified generalization about X*s? 

(xv) A has the following beliefs about astrology. 
Which of the questions below would be root or sig- 
nificant questions that A would have to answer to 
claim her beliefs about astrology were rational? 

(xvi) A refuses to refund a customer's money and, 
when asked, defends her action by stating that it is 
"dictated by store policy". Which of the following 
would be relevant to deciding whether her action 
was indeed "dictated by store policy"? Which of 



the questions would be relevant to deciding if the 
store policy was rational? 

(zvil) Judge A makes the following ruling in a 
case... Which of the following is the clearest state- 
ment of the standards Judge A is using? 

(xvlil) A compares the relation between managers 
and employees to the relation between teachers 
and students. Which of the following would A have 
to answer in order to continue using the analogy 
rationally? 

(six) A gives the following argument for. . .Which rf 
the listed comments would be the strongest objec- 
tion to her argument? 

(xx) Listen to the accompanying excerpt from an 
audiotape of a lecture by A? Which of the following 
questions would be of most help in clarifying A's 
views? 

Multiple-Rating items, Affective Traits, 
(xxl) Here are position-statements from both sides. 
A and B. of a controversial and inflammatory debate. 
From list X below, choose those items which are the 
most reasonable Inferences to draw from position A; 
then choose those items which are the most rea- 
sonable inferences to draw from Position B. 

(xxU) Here are position-statements from both sides. 
A and B. of a controversial and inflammatory debate. 
From list X below, choose those items which state 
the most reasonable assumptions underlying posi- 
tion A; then choose those items which state the 
most reasonable assumptions underlying Position B. 



ERIC 



23 



26 



(xzUl) For each of the items below, tell which Is the 
most reasonable action to take under the circum- 
stances described. If. In your view, there is not 
enough information to make a reasonable deci- 
sion, you may choose the action of suspending 
Judgment as the most reasonable response. 

(xxlv) A disposition to take a measured response 
rather than an exaggerated, dispropoitionate 
response will be measured by requiring students to 
discriminate between the likelihood of dire versus 
mild consequences of positions they dislike. 

Multiple-Rating Items, Intellectual Standards, 
(xxv) The following are four C tuitions from 
Webster's New World Dictionary. Which of them 
gives the dearest definition of...? 

(xxvl) Rank the following definitions for their pre- 
cision on a scale of 1 to 7. 1 means "not precise at 
all"; 7 means "too precise for the subject matter"; 
and 4 means "exactly as precise as it should be". 

(rxvil) Here is a list of data and a series of accounts 
summarizing the data. Which of the accounts Is the 
most accurate summary of the data? 

(zxviil) For each statement below, tell whether it is 
relevant or irrelevant to the hypothesis in the pas- 
sage cited. 

(xxix) Which of the following is the fairest restate- 
ment of the author's position (where the author is 
staling a highly controversial position)? 

(xxx) Rank the following statements according to 
which are the best-evidenced and which are the 
least-evidenced. 

(xxxi) Which of the following Is a good reason for 
believing the statement in question? Which is a 
bad reason? Which is somewhere in the middle? 

(c) Essay Items. 
The full range of the use of critical thinking cannot be 
assessed without requiring writing on the part of the 
student. To confront real Issues, balance competing 
interests, weigh objections and alternatives, and make a 
reasonable decision about a matter of some conse- 
quence— -this Is a major part of what It Is to think critically. 

The ability and the disposition to engage in full-fledged 
critical thinking is measured orJy in part by a person's 
ability to choose from among a pre-selected list. A true 
measure of critical thinking, and thus of a program's 
capacity to Improve critical thinking, can be obtained 
only by including In the assessment generative as well 
as selective dimensions. Neither multiple-rating nor, 
obviously, multiple-choice items are adequate for test- 
ing this dimension. 



Essay items will require proficiency in handling the ele- 
ments of thought in using appropriate macro-abilities, 
in applying Intellectual standards, and, what Is more, 
it will require integrating these and bringing them to 
be vr on a substantive issue. 

Three detailed samples of essay items follow on the next 
page. Each has the same set of general directions. 

In addition to full-blown essay tests, a series of short- 
Justification items are currently being prepared. These 
would not ask students to write an essay on a topic, but 
would rather have them choose an answer from a pre- 
selected multiple-rating list and then Justify their 
answer in a sentence of their own writing. 

This type of test, if it were sufficiently developed, would 
have several advantages: it could be administered, 
because of the brevity and straightforwardness of stu- 
dents' written answers, to the college population as a 
whole rather than merely to a representative sample (see 
(1), under Implementation, below); it would assess 
some, though not all, generative dimensions of critical 
thinking; it would allow flexibility in grading the 
machlne-gradable keyed answers (thus, one could 
adjust the rating of an item up or down depending on 
the Justification); it would be no more difficult to grade 
by trained personnel than the math work on current- 
ly administered standardized calculus tests. 

(3) MTERDISaPIJNAKr AND INTRADISCIPLINARY 
Scope of the Assessment. An assessment of the results 
of critical thinking instruction at the college level ought 
to focus both on thinking within the framework of par- 
ticular academic disciplines and also on thinking in the 
interdisciplinary contexts that are so Important to func- 
tioning as an autonomous, well-informed, productive 
member of a democracy. 

A basic principle of critical thinking Instruction, as 
applied to teaching subject matter in an area, is that (to 
quote the National Council For Excellence In Critical 
Thinking Instruction) "to achieve knowledge in any 
domain, it is essential to think critically". A related prin- 
ciple is that in any domain where one is thinking well, 
one is thinking critically. Any example of good biolog- 
ical thinking, or good historical thinking, or good 
anthropological thinking, or thinking in any other field, 
will necessarily be an example of critical thinking: It will 
involve basic skills dealing with elements of thought; it 
will Involve at least some, and probably many, of the 
macro-abilities; it will involve affective traits luce inde- 
pendent thinking and intellectual perseverance. And as 
far as Instruction is concerned, there Is a real sense in 
which learning biology is learning to think within and 
about the logic of biology. 

Including critical thinking items taken from Individu- 
al disciplines would also properly test those thinking 
skills that are more subject-specific, and it would do so 
in the context of presupposing a good deal of special- 
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Critical Thinking, Problem Solving, 



& Communication Skills Essay Exam 



Hiis test is designed to assess y^uitl-- 
cat^ titfiikkir^^ problem solving and com- 
murrieatiori skills, Yotu? imswer gwlli be 
judged M m clarity, relevance, consis- 
tency* iogte depths coherence* and felo- 
ness* More spedttcally* the reader wffl he 
asking the following questions: 

1) Is the miestlon at issue well stated? 
Is it clear and unbiased? Does the 
expression of the question do 
Justice to tfe coinpleTdty of the 
matter at issue? 

2) Does the writer cite relevant 
evidence; experiences, and/or 
relevant infonnation essential to 
the Issue? 

3) Does the writer clarify key concepts 
when necessary? 

4) Does the writer show a sensitivity 
to what he or she is assuming or 
taking for granted (insofar as those 
assumptions might reasonably be 

. questioned)? 

5) Does the writer develop a definite 
line of reasoning, explaining well 
how he or she Is arriving at his 
or her conclusions? 

6) Is the writer's reasoning well- 
supported? 

7) Does the writer show a sensitivity to 
alternative points of view or lines of 
reasoning? Does he or she consider 
and respond to objections framed 
from 6||fi^ 

8) Does the writer show a sensitivity 
to the implications and/or 
consequences of the position 

he or she has taken? 
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tight to uaethatmoney to advance political oaiia- 
e» :^|jr : bettev^iih^ If ym Mtoti you may decide to 
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tion to the problem and that we have no choice but 
to accept the status quo. 



. issue #3: morality 

Sociologist Irving Oomnan has {K>mted out that all 
tecttve attitude toward members of their 




anyone in such * position. 
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ized knowledge. A critical thinking test In nursing or In 
history of art or In geology might well (In their different 
ways) test for skills of critical observation, while a test 
in sociology might assess thinking skills Involved in con- 
structing an unbiased questionnaire; a critical think- 
ing test in English literature might well presuppose a 
knowledge of who Milton was. while a thinking test In 
physics might Justifiably ask about a problem for which 
a knowledge of the second law of thermodynamics was 
taken for granted. 

Even if we already had a series of critical thinking 
items within the various disciplines, however, we would 
not be testing for marry of the toterdiscipllary abilities 
we most want critical thinking for. Many of these have 
already been mentioned: the ability to make sound 
decisions in the context of understanding our rights and 
responsibilities as citizens. In the context of the work- 
place, as well-Informed and thinking consumers, as 
members of our families, as participants In what is 
becoming a symbiotic and fragile world economy — the 
ability to reason about the gaps between disciplines, the 
bridges between them, and the generallzability of dis- 
ciplines to other areas. 

To test critical thinking abilities— specifically macro-abil- 
ities— as they apply to these areas, what Is needed are 
interdisciplinary questions. These are questions of 
broad interest, ones that shed light on the quality of and 
improvement in student thinking about realistic and 
fundamental Issues; they ought to be the kind of ques- 
tion which can be at least partially illuminated by well- 
integrated knowledge in any number of academic areas. 

The national assessment we are proposing would offer 
a range of mtrodlscipllnary. subject-specific items, 
from which students would choose those relevant to 
their subject-matter knowledge. The interdisciplinary 
items, on the other hand, would not provide choices 
because of the desirability of avoiding the loss of equiv- 
alency that is almost always involved. (That loss would 
have to be minimized to the case of subject-specific 
Items by field testing and rewriting.) 

The interdisciplinary part Is constructable by experts 
well versed In a rich and substantive concept of criti- 
cal thinking. Intradlscipllnary critical thinking assess- 
ment items will be constructed by members of the dis- 
cipline working in consultation with experts In critical 
thinking, perhaps the standing committees on the var- 
ious disciplines of the National Council for Excellence 
in Critical Thinking Instruction. (See Appendix #1.) 

C. The Value of the Proposed Assessment Strategy 

for the Reform of Instruction. 
Since higher order thinking has always been considered 
an important object of post-secondary education, and 
since this assessment would furnish a measure of that 
concept, and sines performance on this assessment 
would have a significant Impact on the standing of 



the college not only In the eyes of the intellectual com- 
munity but in the eyes of the public as well, adminis- 
trators and teachers would have a strong motivation to 
become familiar with the concepts and program behind 
the assessment. Most Importantly, professors and oth- 
ers In charge of instruction and the formulation of 
educational goals would find in it a clear model for the 
articulation and Integration of higher order thinking 
across the curriculum. Note the following: 

1) The concept of the elements of thought not 
only provides a realistic analysis of the 
common dimensions of reasoning in every 
domain, it also encourages the explicit use in 
instruction of those critical/analytic terms 
which are the common possession of the 
Intellectual community (questlon-at-issue, 
problem, evidence, data, concept. Inference, 
assumption. Implication, conclusion, point of 
view, frame of reference, etc.) and makes 
explicit the intellectual standards implicit in 
every discipline as well as in the closer/ 
reasoned professional work In business and 
industry (clarity, precision, accuracy, logic, 
consistency....) 

2) By highlighting reading, writing, speaking, 
and listening as modes of critical reasoning, 
the necessity of having instruction go beyond 
mere didactic coverage of content world 
become more intelligible. As long as reading, 
writing, speaking, and listening skills appear 
the sole province of specialized subjects rather 
than modes of reasoning Intrinsic to the 
construction and mastery of knowledge In any 
subject, there will continue to be a significant 
lack of fit between modes of Instruction and 
modes of necessary learning. 

3) By highlighting the other macro-abilities of 
critical thinking, each analyzed into the same 
elements of thought, there would be significant 
transfer of emphasis on important modes of 
higher order thinking within a larger number 
of college and university student assignments. 
At present many professors fall to notice the 
extent to which they either presuppose that 
students already grasp the nature of 
fundamental Intellectual processes, or they 
make assignments which, though they appear 
to call for such processes, can be successfully 
completed by simply repeating to the professor 
what was said in lecture or written to the text. 

4) By highlighting a common critical/analytic 
language across the disciplines, students are 
encouraged to seek to transfer learning and 
intellectual discipline emphasized in one 
domain of learning to other domains of learning 
and application. The fragmentation of the 
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disciplines, in the minds of the students If not 
in fact, is now a serious problem in higher 
'* education. This problem is mirrored, of course, 
in business, industry, and government in the 
tendency to engage in fragmented, over- 
specialized problem-solving which falls to 
address the macro, multi-dimensional, 
nature of many complex problems. 

5) By highlighting the importance of intellectual 
discipline and grounding it in specific skills 
and abilities, professors and other educational 
leaders will be given a reasonable impetus to 
help students make connections of a broader, 
more interdisciplinary nature. This will also 
be strongly re-enforced by the inclusion of 
everyday, multl togral. toterdlscipllnaiy essay 
questions. 

D. implementation of the Proposed Assessment. 

Our recommendations about Implementation can be 
summarized as follows: 

(I) The essay assessment should be administered 
to a representative sample of the student 
population at each educational institution, the 
machine -gradable items to the total student 
population; 

(II) it should be administered three times during 
a student's college career— at entrance, at the 
start of the Junior year, and Just prior to 
graduation— and thus yield value-added 
Information to institutions; 

(ill) the test should be constructed to be roughly 
three-hours long; 

(iv) test items should be constructed from item 
shells, rather than from a simple pool of actual 
items; 

(v) it should be administered by a private agency 
with critical thinking credentials; 

(vi) it should be paid for by colleges and 
universities that contract to have their 
students tested; 

(vil) it should provide educational institutions with 
detailed Information about central aspects of 
their students' higher order thinking; 

(viil) it should be developed according to the costs 
and timetables listed below. 

Details of our recommendations center around the 
answers to five practical questions about the admin- 
istration of the test: 

(U Who will be assessed? 

Our recommendation is that all portions of the 

assessment be given to, at the very least, a represen- 
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tative sample of the student population at each edu- 
cational institution. Since the problems implicit to 
testing a random sample can be easily worked out, this 
recommendation avoids the expense of administering 
an essay test to the college population as a whole. 

The assessment strategies we have proposed include 
two broad areas of testing: a machine-gradable por- 
tion that includes multiple choice items and multiple- 
rating items and an essay portion. Both portions will 
assess. In their different ways and with their different 
emphases, micro-skills, macro-abilities, affective traits 
and intellectual standards. 

There are, therefore, really two options with respect to 
who is assessed using the strategies we propose. First, 
the machine gradable portion of the assessment can be 
administered to the college population as a whole, 
while the essay portion can be administered to a rep- 
resentative sample of students at each Institution. 
Second, both portions could be given only to a repre- 
sentative sample of the population at each Institution. 
Both options will hold down costs, though the latter will 
clearly be less expensive than the former. Which option 
is ultimately chosen will depend on the amount of 
detail desired, the precise role the assessment is to play, 
and the funds available. 

(W How often will the assessment take place? 
The maximum benefit to educational institutions will 
be provided to the extent that they are enabled to mea- 
sure the progress of their students' higher order think- 
ing during the course of their college career. This will 
enable Institutions not only to gauge their contribution 
to their students' progress, but also to measure the suc- 
cess of attempts to re-design their instruction so as to 
increase critical thinking capabilities. 

These objectives can be accomplished by having stu- 
dents assessed often enough to reflect such progress, 
optimally: at the time of their entrance, at the beginning 
of their Junior year, and Just before graduation. 

(HQ How long will the test take? 
The test should last about three hours in order to cover 
multiple-choice, multiple-rating, and essay items without 
becoming a speeded test to an inappropriate degree. To 
span all difficulty levels, it would be best to have a total 
of at least 30 items. While two of these could be short essay 
items requiring 20 minutes each to answer, the machine- 
gradable items would be faster to answer, and hence 
could be handled in 3-8 minutes. 

fw) How will a MuffLciently large pool of items be 
constructed? 

While it might be possible to release a pool of items which 
would provide the equivalent of 6 tests, hence 6 x 30, it 
would be better to Increase flexibility by using item shells, 
which would be items that Include identified variables, 
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each of which could be replaced from a list of acceptable 
values. This would greatly Increase the number of items 
that could be generated, but without "surprises". A pool 
of shells would generate over a thousand items, possibly 
several thousand. 

(v) Who will do the assessing? 

In order to avoid problems in the reliability of the assess- 
ment (like those we have seen occur in the California 
Direct Writing Assessment), the assessment needs to be 
monitored, administered, and graded by a private agen- 
cy whose personnel r iave critical thinking credentials or 
are at least under the direction of scholars with a solid 
grounding in ^search in critical thinking. 

(vi) Who will bear the costs of the assessment? 

The assessment should be paid for by the colleges and 
universities that contract to have their students test- 
ed. This not only puts least burden on the public but 
represents an established precedenfin distributing 
costs of testing. 

(vii) What will institutions be able to learn from the 
results of the assessment? 

We anticipate that colleges and universities will receive 
an analytic report that will document all of the follow- 
ing: 

• where their students are strongest and weakest 
with respect to particular microskills; 

• where their students are strongest and weakest 
with respect to important macro-abilities; 

• how students stand in each of the college's 
majors; 



• how their students stand in relation to 
students at other institutions; 

• how their graduates stand in relation to their 
Juniors and their entering freshmen; 

• how their students stand with respect to 
established performance criteria. 

This information would enable institutions to target 
instruction to remediate weaknesses and build on 
strengths, as well as to measure what students are gain- 
ing as a result of attending their classes. 

hriii) What is a reasonable estimate 
of the cost of and timetable for developing 
th« national assessment? 

It would be possible to develop the most restricted ver- 
sion of a series of three parallel tests — for entrance, 
Junior year, and preceding graduation — in nine months 
at an estimated cost of $240,000. This version would 
be restricted a) to using only fully articulated items, 
rather than the more flexible pool of item shells, and b) 
to using only interdisciplinary items. However, rescric- 
tion to three fully articulated forms would be useful only 
if security for the test were possible. In current contexts, 
especially New York State, it is difficult to maintain test 
security against the legal demand for full disclosure to 
facilitate legal hearings on protested results. 

The full assessment in its most desirable form, includ- 
ing both subject-specific items and the pool of item 
shells described in (iv), would involve seat time to 
develop, and would then be subjected to expert criti- 
cism, rewrite, and re-criticism, and to two rounds of field 
testing with intervening rewrite. It could be done in two 
academic years, at an estimated cost of $350,000.* 



* The authors wish to acknowledge the invaluable 
advice provided us by Michael Scriven on evalu- 
ation theory in general, and, more particularly, on 
the logistics of test construction. 
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Appendix #1 

National Council for Excellence in Critical Thinking Instruction 

Standing Committees 

Membership in the following standing committees is being established. Membership Is limited to individuals 
who have special expertise in the academic area delimited by committee name* 



Critical Thinking and Amtmmcnt 

Critical Thinking Standards 

Critical Thinking Tests 

Critical Thinking Assessment 

Critical Thinking and the Assessment of Education 

Critical Thinking and the Evaluation of Teaching 

Critical Thinking and Basic Skills 
Critical Thinking and Reading 
Critical Thinking and Writing 
Critical Thinking and Listening 
Critical Thinking and Oral Expression 
Critical Thinking and Reasoning 
Critical Thinking and Media Literacy 

Critical Thinking In thm Disciplines 
Critical Thinking Across the Disciplines 
Critical Thinking in Mathematics 
Critical Thinking in Science 
Critical Thinking in Hlstoiy 
Critical Thinking in Sociology 
Critical Thinking in Anthropology 
Critical Thinking in Political Science 
Critical Thinking in Social Studies 
Critical Thinking In Language Arts 
Critical Thinking and Rhetoric 
Critical Thinking and Psychology 
Critical Thinking and Cognitive Psychology 
Critical Thinking and Philosophy 
Critical Thinking In Nursing 
Critical Thinking in Home Economics 
Critical Thinking in Vocational Education 
Critical Thinking In Business Education 
Critical Thinking in Communication Studies 
Critical Thinking in Legal Education 
Critical Thinking and the Arts 
Critical Thinking in Religious Education 



The Nature and Theory qf Critical Thinking 

Critical Thinking and Informal Logic 

Critical Thinking and Creativity 

Critical Thinking and the Understanding/Assessing of 

Assertions and Questions 
Critical Thinking and Developmentallsm 
The Role of Reasoning in Education and Critical Thinking 
The Role of Affect In Critical Thinking 
Critical Thinking and Moral Education 
Monologlcal and Multllogical Thinking 
Critical Thinking and Practical Epistemology 
Critical Thinking In the Assessing of Knowledge as Design 
Critical Thinking and Practical Reasoning 
The Role of Critical Thinking in Broadening and Assessing 

Points of View 

Critical Thinking and the Recognition and Understanding 
of Ignorance 

Critical Thinking and the Recognition of Common Mistakes 
in Reasoning 

Critical Thinking and Ideology 

Critical Thinking and the Art of Questioning 

Critical Thinking and the Role of Images in Thinking 

The Hlstoiy of Critical Thinking 

Critical Thinking Pedagogy 

On the Fostering of Critical Thinking In Young Children 
Critical Thinking and Remedial Instruction 
Critical and Multi-Cultural Thinking 
Critical Thinking and Computer Assisted Instruction 
Critical Thinking and Cooperative Learning 
Critical Thinking and Educational Policy 
Critical Thinking In Accreditation and In the Baccalaureate 
Developing a School Environment Conducive to Critical 
Thinking 

Critical Thinking Staff Development 
Critical Thinking and Learning Centers 
Critical Thinking and Preservtce Teacher Education 
Critical Thinking and Minority/Ethnic Issues 



Critical Thinking and Educational Level* 

Critical Thinking and Elementary Education 

Critical Thinking and Middle School 

Critical Thinking and High School 

Critical Thinking and The Community College 

Critical Thinking and The Four-Year College or University 
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Appendix #2 
Critique of Student Essay from CAP 



The student essay entitled "Rock Around the Clock", if 
graded by those with a background In critical thinking 
and reasoning would. In the professional Judgment of 
the authors of this paper, have been graded at the 
lower rather than the higher end of the continuum of 
eight levels: "minimal evidence of achievement" or. at 
best, "limited evidence of acnlevement" rather than 
the highest grade of "exceptional achievement". For 
though the essay may have "flair and sparkle" (as one 
teacher expressed It), it is a poor example of evaluative 
reasoning, since It systematically confuses the objec- 
tive goal of reasoned evaluation with the very different 
goal of explaining subjective preference, an impor- 
tant distinction in critical thinking which the teacher- 
evaluators apparently missed entirely. 

First of all the instructions themselves are confused. They 
begin with a clear requirement of "objective" evaluation: 



Students were asked to write an evaluative 
essay, make Judgments about the worth of a 
book, television program, or type of music and 
men support theirjudgments with masons and 
evidence. Students must consider possible criteria 
on which to base an evaluation, analyze their 
subject in the light of the criteria, and select evi- 
dence that clearly support? theirjudgments. 

Unfortunately, this request for reasoned evaluation is 
blended in the second half of the instruction with what 
might possibly be taken, with a little stretching and 
selective reading, as a request for the expression of a 
"subjective" preference: 

Each student was assigned one of the follow tng 
evaluative testes: to write a letter to a favorite 
author telling why they especially liked one of the 
author's books, to explain why they enjoyed 
one television program more than any others, or 
to Justify their preference for a particular type of 
music. The tasks made dear that students must 
argue convincingly for their preferences and not 
Just offer unsupported opinions. 

Let's look closely at this confusion. In the first place, 
there is still an emphasis on objective evaluation ("The 
tasks made clear that students must argue convincingly 
for their preferences and not Just offer unsupported 
opinions") at the same time that the task itself is 
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defined both as an "evaluative task" and as a Justifi- 
cation for a "preference". 

Now most people prefer books, television programs, 
and types of music for fundamentally subjective, not 
objective, reasons. They like a particular book, television 
program or song for no reason other than that they do 
Wee it, that Is. because they enjoy it or find pleasure in 
it or are interested or absorbed or excited or amused by 
it Each of these affective self-descriptions is typi?alty not 
the result of an objective evaluation. They have no rela- 
tion to the objective quality of what Is Judged. They are 
about the personal responses of the experiencer, not 
about the objective qualities of that which is experienced. 

Most people, to take the point a step further, do not have 
"evidence"-— other than the stuff of their subjective 
reactions— to Justify their preferences. They prefer 
because of the way they for not because of the way they 
reason. To choose because of these subjective states of 
feeling is precisely to lack criteria of evaluation or evi- 
dence that bears upon objective assessment. When 
challenged to support subjective preferences, people 
usually can do little more than repeat their subjective 
reactions ("I find it boring, amusing, exciting, dull, 
interesting, etc...") or rationalize them ("I find it excit- 
ing because it has a lot of action in it"). 

A reasoned evaluation of a book, a program, or a type 
of music requires more than this; it requires some 
knowledge of the qualities of what we are evaluating and 
of the criteria appropriate to the evaluation of those 
qualities. One needs to be well-informed about books, 
about programs, about music if one is to claim to be in 
a position to objectively evaluate them. If one is not well- 
informed one is unable to render a Justified evaluative 
Judgment, though one can always subjectively react and 
freely express one's subjective reactions as (mere) per- 
sonal preferences. This Is what the student (graded as 
having written an objective evaluation of "exceptional 
achievement") actually does. But his evaluators. not 
having this distinction clear In their own minds, com- 
pletely miss the difference. 

The model student essay can. for analytic purposes, be 
divided into three parts. We shall comment briefly on 
each in turn. The first segment of the essay Is an 
account of a highly emotional exchange between the 
student and his mother 



"Well, you're getting to the age when you have to learn to 
be responsible!" my mother yelled out. 
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"Yes, but I can't be available all the time to do my 
appointed chores/ I'm only thirteen! I want to be 
with my friends, to have Jim! I don't think that It 
is fair for me to baby-sit while you run your lit- 
tle errandsr I snapped bade I sprinted upstairs 
to my room before my mother could start anoth- 
er sentence. 

It Is clear that In this segment there is no analysis, no 
setting out of alternative criteria, no clarification of 
the question at Issue, no hint at reasoning or rea- 
soned evaluation. 

In the second part the student makes a sweeping claim 
about a purported causal relationship between listen- 
ing to rock music and his asserted, but unsupported, 
ability to control his emotions. He does not consider 
"possible criteria on which to base an evaluation". He 
does not present any evidence, though he does cite two 
examples, one where a song prompts him to punch his 
pillow and one where another song prompts him to stop. 
This gives little credence to the notion that rock music 
leads to his "controlling" his emotions. If anything, his 
examples seem to imply that, rather than learning 
control from, he is learning to be controlled by, the 
music he listens to. His major claim, that "Without this 
music, I might have turned out to be a violent and 
grumpy person," Is without reasoned or evidentiary 
support. He merely brashly asserts that it is true: 

I turned on my radio and "Shout'' was playing. I 
noted how true the song was and I threw some 
punches at my pillow. The song ended and 
"Control*, by Janet Jackson came on. I stopped 
beating my pOlow. I suddenly felt at peace with 
myself. The song had slowedme down. I pondered 
briefly over all the songs that had helped me to 
control my feelings. The list was endless. So is my 
devotton to rock music and pop rode These songs 
help me to express my feelings, they make we 
wind down, and above all they make me feel 
good Without this music I might have turned out 
to be a violent and grumpy person 

In the third, and final, section of the essay the student 
closes his remarks with a series of subjective, unsup- 
ported, even irrelevant statements: 

Some of my favorite songs are by Howard Jones, 
Pet Shop Boys, and Madonna. I especially tike 



songs that have a message In them, such as 
"Standby Me, by Ben E.King. This song tells me 
to stand by the people I love and to not question 
them In time of need. Basically this song ts telling 
me to believe tn my friends, because they are my 
friends. 

My favorite type of music Is rock and pop rock. 
Without them, there is no way that I could sur- 
vive mentally. They are with me tn times of trou- 
ble, and best of all they are only a step away. 

If this is reasoning, it Is very bad reasoning: "Believe in 
your friends because they are your friends", "If you feel 
you cannot survive without rock music, then It follows 
that you can't." Of course, a more appropriate ' fcr- 
pretatlon of what Is going on is that the student is not 
reasoning at all but merely asserting his subjective 
opinions. Consider, the student doesn't examine alter- 
native criteria on which to base an evaluation of music. 
He doesnt analyze rock music in the light of evaluative 
criteria. He doesn't provide evidence that clearly sup- 
ports his Judgment. His writing Is vague where it needs 
to be precise, logically rambling where It needs to be crit- 
ically reasoned. We dont really know what he means by 
songs "controlling" his feelings. We art* not provided with 
any evidence on the basis of which we could assess 
whether there is any truth in his sweeping claims 
about himself, for example, that he could not survive 
mentally without rock music. Indeed, common sense 
experience strongly suggests, we believe, that the stu- 
dent is simply deluding himself on this point, or, alter- 
natively, engaging in unbridled hyperbole. 

We are prepared to be sympathetic to students who 
don't understand the difference between reasoned dis- 
course and subjective assertion, but we cannot be 
sympathetic to the national dissemination of fluent 
subjective reactions as a model of good reasoning and 
rational evaluation. The damage that follows from such 
an ill-conceived model is Incalculable. 

When a blatantly weak essay is disseminated as an 
example of "exceptional achievement" in the writing of 
a reasoned evaluative essay— with accompanying direc- 
tions calling explicitly for consideration of alternative cri- 
teria, analysis In the light of (appropriate) criteria, pre- 
sentation of evidence that clearly supports conclusions 
drawn— then it Is clear that a non-substantive concept 
of critical thinking and reasoning Is at work. 
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A Review of Richard Paul and Gerald Nosich's "A Proposal for 
the National Assessment of Higher Order Thinking at the 
Community College, College, University Levels" 

Lor en z Boehm 
Oakton Community College 

I like this essay. I find much in it with which to agree; 
I'm impressed by its careful, thorough, and respectful 
exploration of what critical thinking is and how it might be 
assessed; and I appreciate what it contributes to any conver- 
sation about the shape of a national assessment test. Thus, my 
strongest reaction is applause. 

However, while there is much to agree with and be pleased 
by, there are a few points that I have questions about, or that 
trouble me, and that I would like to lift out and set aside, if 
only to save them for discussion at another time. 

One among these is the "assessment strategies" professors 
Paul and Nosich suggest. I have a lot of trouble with multiple- 
choice tests. They seem to me, in a way, intellectually 
dishonest. Give a student a piece of prose to read, and then ask 
her to write-out state the main idea of the piece. I like 
that. It tests her understanding. If, on the other hand, you 
give her a list of possible main ideas and ask her to choose the 
correct one, you may be assessing her understanding; more likely 
you are assessing her ability to identify the correct answer, 
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which she may not have known on her own, but which you have given 
her, and which she now arrives at by eliminating the incorrect 
ones — all of which, to my mind, is a different kind of 
"understanding" altogether. In the latter, there is a built-in 
crutch (and that's the "dishonesty"); further, it doesn't assess 
the same mental ability, although it pretends to. It also opens 
the door to/for guessing, which is intellectual dishonesty of 
another sort. 

Now, I see that the kinds of multiple-choice questions Paul 
and Nosich have in mind are more complex, but I believe the 
"dishonesty" I described is still there. As long as there is a 
choice of answers there is a crutch; the test is doing some of 
the work. As a result, I would eliminate multiple choice 
questions of this type. I would much rather have the student 
formulate her answer out of her interaction with the piece she is 
being questioned on. I think in the jargon this is called a 
"free response" test item. In any case, I believe makes for a 
better test. 

I'm not sure if the same can be said about "multiple-rated" 
test items. I see that these allow for different kinds of 
questions. And I see, as the authors point out, that "guessing, 
using the process of elimination, and scoring well because of 
test-taking skills are all but impossible." Still, I prefer test 
items that ask students to generate text. In fact, the example 
in figure 4 (p. 21) would, it seems to me, be ideal if it asked 
students to do what Paul and Nosich do: choose the answer from 
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the alternatives (EC, UA, or N) and then explain in prose why 
that choice was made . Such a test not only shows that the 
student can reason out the "correct" answer; it also shows the 
reasoning process the student went through which, in my opinion, 
makes it a superior test. 

I do understand that there are other considerations about 
multiple-choice tests. They are easier and faster to score, 
especially if the scoring is done by a machine. Fine. Then the 
issue is efficiency or speed, not the nature or the efficacy of 
the assessment. They are different principles and, at the risk 
of sounding naive, I'll say that the efficacy of the assessment 
should get the higher "score." 

Needless to say, I am most supportive of the "essay items" 
ni.ofessors Paul and Nosich discuss. While I'd like to talk with 
them a little about the shape of the prompts, here we have almost 
no disagreement, save for how big a role writing should play. As 
I maintained in my review of Ed White's essay, I believe it is 
the superior method of asssessing critical thinking and should be 
the primary (if not the only) basis of its assessment. In the 
context of this review, I would simply add my belief that Paul 
and Nosich are exactly on target when, in the "Objectives" 
section of their essay (#11, p. 3), they argue for an assessment 
test that "is empowering" and that "promotes. . . 'the active 
engagement of students in constructing their own knowledge and 
understanding'." I believe there simply is no better way of doing 
that than through self (student) -generated prose written in 
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response to a carefully crafted prompt. 

A second point that professors Paul and Nosich make that I'd 
like to explore further has to do with the distinction they draw 
between "interdisciplinary and intradisciplinary. " This is, 
afterall, one of the central dichotomies of the critical thinking 
movement and bears some talking-about . I like the idea proposed 
here that 

An assessment of the results of critical thinking 
instruction at the college level ought to focus both on 
thinking within the framework of particular academic 
disciplines and also on thinking in the interdisci- 
plinary contexts that are so important to functioning 
as an autonomous, well-informed, productive member of a 
democracy, (p. 24) 

Some faculty members at some colleges argue that critical 

thinking is best understood as a set of mental abilities (like 

the ability to construct, analyze, and evaluate arguments; or the 

ability to apply, analyze, evaluate and synthesis ideas) , and 

that these are best taught in separate critical thinking courses. 

Others argue that it is best understood as doing the mental work 

of the disciplines, and is best taught in all courses across the 

curriculum by teachers who have "unpacked" the thinking required 

by their discipline, organized it gradiently, and have made it 

the backbone of their courses. However, most critical thinking 

teachers now agree that doing both is optimal. (This is decidedly 

the case at my own institution, where for six years faculty 

worked successfully to build the " intra disciplinary" model, and 

where recently they have begun to develop a course in the 

"interdisciplinary model, which is now seen by them and just 
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about everyone else as a compliment to what came before and 
continues . ) 

The move to include assessment items that address both 
models is, I believe, a very strong aspect of what Paul and 
Nosich recommend. Conceptually. However, if the idea fails to 
win support, then I would argue for a test that assesses critical 
thinking defined as understanding and applying at the appropriate 
gradient the modes of inquiry, the language, the thinking, done 
by practitioners of the disciplines. 

A third point. I'm interested in the idea of assessing what 
Paul and Nosich call the "affective dimensions" of critical 
thinking; I also appreciate the difficultly in doing it that they 
forsee. As they say, "For some of these affective dimensions 
(intellectual perseverance, for example) , any testing would have 
to take place over an appropriately long period of time and thus 
[they] could not be legitimately assessed at all during the time- 
frame suitable for a national test." Perhaps. Without, for now, 
addressing the value, or the lack of value, of assessing 
"thinking independently," or "intellectual perseverance" or 
"intellectual courage," or any of the others, it seems pretty 
clear to me that these can be assessed very effectively by 
portfolio, which allows for, even calls for, a variety of 
assessment materials, including, especially, drafts of essays on 
a range of topics, which reflect the student's thinking process 
as well as his disposition toward thinking. 

One last area of concern. Among the "main objectives of a 
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process to assess higher order thinking skills," the authors 

identify this one: 

It should respect cultural diversity by focusing on the 
common-core skills, abilities, and traits useful in all 
cultures. (#11, p. 3) 

I think I understand the intention here, but I also think there 

is a dilemma. I don't believe test designers can (or should) 

determine which ai-e to be the "common-core skills, abilities, and 

traits useful" in other cultures; it seems best to let other 

cultures do that piece of determining. What can be done is 

identify "the skills, abilties, and traits useful," even 

necessary, for succeeding in £&is "culture," whether broadly 

defined as this "society" or more narrowly defined as a 

particular workplace, or an academic discipline. I think Paul 

and Nosich are trying to put the best light on the unavoidably 

dark side of any national assessment of critical thinking. There 

have to be standards of some sort; they may not be everyone's 

standards, predictably they won't be; certainly they won't be 

every culture's standards. Some won't like that. It's 

unfortunate, but I don't think we should avoid seeing it for what 

it is. 
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A Critique of Richard W. Paul's and Gerald M. Nosich's 

"A Proposal for the National Assessment of Hlgher-Order 
blinking at the Community College, College, and 
University Levels* 



Prepared by Peter A. Facione 

A figure of international renown in the Critical Thinking Movement, Richard Paul 
m collaboration with Gerald Nosich, offers a detailed, nnUMu^s^ 
and pronouncements with regard to critical thinking and critical tMnkina a^ss^ent Th* 
baccalaureate level. TTieir paper focuses on three chief ™esC^ 
by which a national assessment program should be evaluated? What does critical S 
S; ni j at ™ ite com P<> n f t parts? And, what about assessment instrameSn? 
The first question is normative, the second conceptual, and the third empirical. This review 
addresses each question in turn. H 

rvit,v fl i D ^wL^ W if )P ! ,l Sl ! wh ° sc vision produced the Center for 

Critical Thinking and Moral Critique and more than a decade of international CT 
conferences, begins by suggesting 21 criteria upon which a national critical thinkina 
assessment program might be evaluated. Addressing this issue at this level of detail is, in 
itself, a positive contribution. In so doing, Dr. Paul forcefully reminds us that there are 
many things we must keep m mind if we are going to do this job well. 

Are Dr. Paul's 21 the right 21? How do his %\ relate to what the experts in 
educational assessment would advise? Is each of the 21 clear, operational, and free from 
questionable assumptions? Is each expressed at the correct level of abstraction? Is the set 
of 21 comprehensive and reasonable? Which take priority over others? 

A brief look at only one of these 21 offers an example of the many concerns each 
provokes and amplifies the importance of working with the experts in educational testing 
and CT testing to develop the proper set of criteria. #1 says the CT assessment process 
should assess students' skills and abilities in analyzing, synthesizing, applying, and evaluating 
information.^ The positive value of this proposed criterion is to point us toward content 
validity, a vital component of any sound assessment design. Content validity put more 
abstractly (as it is in the research literature of educational testing) asks, "Is the theoretical 
construct, X, which this test instrument targets the proper one? Unfortunately Dr. Paul's 
way of putting criterion #1 compresses the theoretical concern for content validity with a 
partial list of some CT skills. A well-formulated criterion would separate the theoretical 
consideration (content validity) from an incomplete analysis of that content. A revised #1 
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might read "...should target an appropriately rich conceptualization of CT. H 

One could challenge and revise each of Dr. Paul's proposed list of 21. 1 But would 
working through Dr. Paul's proposed list in a detailed, critical way advance our common 
goals? No. To his credit, Dr. Paul has aimed us in a useful philosophical direction: We 
must establish the criteria - both in general and in detail by which we will evaluate any 
proposed national system of CT assessment. 

To carry out this part of the task we should turn from philosophical speculations to 
the technical advise experienced scholars in the field of educational tests and measurement 
can bring to the table. We should contact the likes of Robert Ennis, Stephen Norris, Joanne 
Carter-Wells, and Barbara Lawrence, as well as other experts in the psychological science 
of educational assessment, and invite them to carry out the philosophical direction set by 
Dr. Paul. These experts should be invited to review the scientific literature on educational 
assessment and identify the criteria of a suitable national CT assessment program. Thus 
content validity, along with construct validity, concurrent validity, reliability, etc could be 
identified as appropriate general criteria without begging any questions with regard to how 
those various criteria play out in the case of critical thinking. 

Experts experienced in the technical aspects of CT test validation should be asked 
to advise on how each general criterion should apply to CT assessment. For example, in the 
case of CT assessment, construct validity (that the process that the test-takers must use to 
achieve the correct answer is, indeed, the process which the test purports to assess) is 
extremely complex. 2 Stephen Norris and Robert Ennis are very helpful in this regard, for 
in their book Evaluating Critical Thinking they provide a useful and highly readable list of 



1 For example: Aren't "maximum flexibility* in #2 and 'Important differences" and 'crucial to all the disciplines" in #3 
dysfunctionally vague? What does #4 really mean, conceptually or operationally? In #5, do "readily lead to improvement" (p-3) and "can 
be wed to lead to improvement" (p. 5) mean the tame thins? Am the ephtemolofkal aseunptions inherent in #6 really true? In #7, 
what am -versatile" ikilk and do they differ from "fundamental" skills? Also, Ooeao't "lespontiMe, decision-making" in #7 open the door 
to a whole series of moral judgements which take us «ell beyond critical thinking BILK? k>1 #1 a meta-crit*rioa, when compared to 
the others? How are "adult level* in #9 and "college level" ie #12 related? Is the strategy proposed in #10 the right way to wjhievc the 
toal cited in #10, and shouldn't a criterion propose the goal only? How an #11 and #3 related: la #11 content vaUdj are its pedagogical 
assumptions comet? Does the theoretical basis for #12 accord with contemporary reaemth in the areas of reading and writingse rneanini 
makinipnxesses7 Isn't It the case that some of the "central" things cited #13 go beyond critical thinking? What doee "aut*oticnlly 
usable" M^nTmean and what is 'nductionism"? Does "bask at the college level" in #13 nwan remedial or something else? Does the 
"Urge body of the populace* in #16 rule out the right idea If perceived only by a small mtmvity (of experts, say)? In w»ut "«y does 
-vafcable skills thatapply to genuine problems" in #16 overlap with critical thinking? la the sptdflc assessment strategy eked in #17 a 
rithtonc and is it thaonly right one? Isn't #1? more of an implementation tuggeetion than a » si criterion whereby to judge a national 
am^t^^T Whit sped*, things am the imendedcon^StT"r*al Ufa problems" in #117 In #W. ".ffontabk" to whom - 
uH^tlo^sttte or fede^goveitiment, individual teet-takers, potential employers? Why should It be a criterion , of thenatJonal 
n^T^tZtouTt^^t way #20 suggests to inetttbtlonal evaluation? Is CT testing achievement testing (aa in how much 
O ^^J^ as aptitude testing (aa in what ere the levels of one's skills)? Should #21'. 

o^^.!£dm^ *+* •* «* «~A**»" 

HBaEBBflB educational attainment be differentiating factors or should there be only on* national standard? ^ n 
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criteria by which to evaluate the quality of a CT assessment program. 3 Additional criteria, 
specific to CT assessment, should be added. 4 Fiscal and political criteria specific to the 
project for which this work has been commissioned should be identified and, if they do not 
violate the technical criteria, also be added. Criteria should not be confused with strategies 
for implementation, nor should they preempt the findings of empirical investigations. 



A second useful contribution of Dr. Paul's paper is the attention he gives to the 
concept of critical thinking. His main point is that, yes, we do have a rich, multi-textured 
conceptualization. There is no need to ponder anew what critical thinking might mean. 
There is a consensus among CT experts about core CT skills. There is accord about the 
dispositions, or habits of mind, associated with good critical thinking. We can advance to 
the next step, which is the practical issue of how to assess these skills and these dispositions 
as they are, or should be expected to be possessed, by baccalaureate prepared persons in 
our society. 

Dr. Paul cites a draft statement prepared for his newly formed CT coalition, the 
National Council for Critical Thinking. The draft happens to reinforce and largely confirm 
that conceptualization of core critical thinking skills and dispositions which emerged from 
the work of the national Delphi research project, conducted during 1988 and 1989 under the 
auspices of the American Philosophical Association. 

The Delphi research project adopted a qualitative social science method developed 
by the Rand Corporation, known as the Delphi Method. Carefully conducted rounds of 
questioning, argumentation, refinement, and reformulation led to a consensus among a panel 
of 46 na: tonal experts (including Dr. Paul) regarding the core elements in the concept of CT 
which should be expected at the college level. The Delphi Panel took up many issues during 
its two years of work; it considered a variety of views. One of the panel's must useful points 
of consensus - reprinted at the end of this review - is the detailed a definition of each core 
CT skill and sub-skill, with examples of educational outcomes. In these outcomes many see 
examples of the kinds of tasks that might also be used in a comprehensive CT instruction 



3 Stephen P. Norm end Robert H. Ennls, B«iu.ti^ rritteel Thtnkine. Midwest Publications, Pacific Oiove, CA, 1989. 

* CT teste should presume a level of cognitive development appropriate to the subjects to be tested (and should not assume 
that colleie students, for exempk, will necessarily approach problems the way expert togkieno or scientists orpwgremmen j»£t)CT 
teste should presume no technical CT vocabulary. Comet answers should not be dependant upon Information recall. CT t^ «bou4d 
require that subjects use CT, rather than remember things scholars might ay about CT, to achieve comet responses. Although explanation 
IslcoraCTsUuTeomct answers to soma items may be achieved through the proper appHc^Jon of other, more prsllmlnery critlcel thinking 
skills, such asailalyais or inference. Those being tested should not necessarily have to be abte to explein the processes whereby they 
correctly appUed mote preliminary skills. However, other turns in e complete CT assessment program should target «be criUriotegicel, 
ZSZS^vSuS^Zi concept eoneiderntione which are involvd In en eiptenetion eulteble of ~»l O^^critk-^Un^ 
^V^Xw!^mJmC Doe. No: TM 012 917). 'Strategies for Multiple Choice CT Aaaeesment,* n ^ftfteyind 
* ^^STm Vale Press. Newport News, VA. 1991. snd Thirty W.ys to Meu Up « CTTm,' i«*w»t Vol. 
kJ&&L 12. No, 2, pp. 106-112, Spring 1990. 
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and assessment process. 3 

The Delphi research consensus produced a list of six core critical thinking skills to 
be expected at the college level: analysis, interpretation, inference, evaluation, explanation, 
and self-regulation. At the same time, the expert panel identified a set of critical thinking 
dispositions that characterized how a good critical thinker approaches life and lrvingin 
general and specific problems or questions that might arise. In so doing the Delphi panel 
drew some instructive distinctions. Among the most significant distinctions was that dr vn 
between the procedural, laudatory, and normative uses of the term "CT". In other words, 
a key question the experts resolved was how many main parts does CT have? First, does 
the concept of CT include cognitive skills? Second, does it also include affective 
dispositions? Third, does it include a moral component? There was no doubt about the list 
of cognitive skills. What the six are and that they are part of what is meant by "CT was 
solidly accepted. But some experts, particularly those from the Center for Critical Thinking 
at Montdair State, argued that dispositions were not part of the meaning of "CT. However, 
The Executive Summary of the Delphi research expresses the majority and minority views 
on this issue as follows: 6 

* 

The experts are in consensus regarding the list of affective dispositions which 
characterize good critical thinkers. However, whether or not these affective 
dispositions are part of the meaning of "CP in the way that the cognitive 
skills are, was an issue which divided the experts from the first It became 
evident that various experts mean different things when they used the term 
"CT in reference to its possible dispositional components. 

The deepest division is between the nearly two-thirds majority who hold that 
the term "CT includes in its meaning a reference to certain affective 
dispositions and the roughly one-third minority who hold that "CP refers only 
to cognitive skills and dispositions, but not to affective dispositions. The 
majority (61%) maintain that the affective dispositions constitute part of the 
meaning of "CT." They argue that these dispositions flow from, and are 
implied by, the very concept of CT... These experts argue that being adept 
at CT skills but habitually not using them appropriately disqualifies one from 
being called a critical thinker at all. Thus, in addition to using "CT in its 
procedural sense, these panelists also use h CT in its laudatory sense. They 
find it sensible to say, "litis person is a critical thinker, but this other person 
is so mentally lazy, close-minded, unwilling to check the facts and unmoved 
by reasonable arguments that we simply cannot call him a critical thinker." 



5 TABLE 4 in particular, Of Critical Thlnklnr A Statement of Email Comemui for Purpoiaa of BmUmfJ Amifffltllt 
and Initruction. (ERIC Doc. No: ED 31S 423). 

* n&wcuilve Summary of Critical Thinkinr A Stotcmunt of Bmart Co—mm for Pumoi« of EftraiiMH AHMBBMB llttl 
^ Initmclion. * California Academic Preat, 217 La Cruz Ave., Millbrae, CA, 1990. 
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the proper affective (tor^htenl v^iu .K.l.^ n J because *« P 8 "™ <*» 
parting the^r^Xlteti,? <oZ^^ ri ^ d<K » uem 

critfejif thhtL-mmm tw . cnucai tmnktng from what is true of sood 

person, because of his CT skflli should bTcal M^SS^t L?"J SU . ch ! 
a good one, (in terms of his effective use ohhose " ^ 

progran^aSreS CT^^*!^,^ ^J*"" 1 out, a full assessment 

As suaestcd above, there are two senses of the term "good- which might be 

to the thinker's effectiveness and responds to the question, "How well is his 

responds to the question, "Is this person's use of CT ethical?" TTwMMeof 
"good" the experts intended became clear: 
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nenon U doing, Whil "CT **»**> why It Is or value, anu im wu* 
rented .. three distinct concern.. 

Dr.Pau.has^«edusintwou*f^^^^^^ 
us of the need lo define with great F^^^ri J™** cfLessment. 

ofcomments.pron^cement^ they do pot 

or cannot be used successfully tottsess ^ stu dies. At several points 

support their opinions with ex^nmen^ v^d^ o _ fe ^ new ^ 

the?suggestfbrnatio,^a^th^ canonize folk wisdom 

calling it experimental jvou d be pronator. *^ e ' technica l topics. Hence, to 

The key piece of «^?<?ZZr L^nX^l^ 
assessment might >«^XTc ^ti^tob^lved by experimental research, 
fit which aspects of CT best *a ^"^^^phMca.ed appUcation of several CT 
Whether a certain «|^' a «T™' to or that modality is an experimental 
skills, can be tested «^ Squestions, just as there is no a prion answer 

question. There are no a pnon answers »*J"S^ B 2J i ^ found to provide mobile 
I the question "Can • ^ XS™<*" earlier era confidently to declare 
telecommunication capabilities? For p lwosop n of OTer mo u ntaln , 

that we could not do this because ^c«cannoUW £ g technica i experimen«l 
was the same kind of mistake - it was an attempt <»y ^ ^ o( experimental 
question with an « prion ' P m ^"f CT ttJ cT^C^^*'th a multiple-choice 
evidence, that we can't effectively '«« ^IU »d „ v ^ d ^ 

instrument is equally Pfp^^^frfhS-^cognltive skills and affective ti»P«W on ? 
^^^^To^tb^ community. Given a suitably rich 

' ^ ^-n** , M that either it luch t**t iMiru reduce 

,o m«lii|*MN*- ^(^»««oti«^^S^ for 



conceptualisation of CT, let's invite empirical research, based on sound rwvdmi™,™! 
principles, which addresses the scientific question of howtoTst meas^CT? 

In summary, Dr. Paul is right to call for the articulation of a comprehensive set of 

inrougn the Delphi research project CT exnem hnJ» ~™ ^ ^ u 
conception of CT, richly tex^^d^LpTe^uuTin'Z o^cogni~ 
and affecuve dispositions (habits of mind), which can serve to guide ^JSZ 
and ground our concerns about content validity. TOrd Dr. Paul raises tenwrtanK,^ 
regard to assessment strategies. Experts in educational testing and we^uTtfCT 

S«« T^ spe *% *" * ^ of em PWcal research aT^enXe^rten^ 
should be called m to address technical concerns regrading criteria and strategic 



From the Delphi Report 
TABLB4 - CONSENSUS DESCRIPTIONS SIX CORE CT SKILLS AND SUB-SKILLS 

1.1 CATEGORIZATION: 

toSnKt^^ OTaPP, ^ ,ely f0nBMta,e "•"T^ dtetlnctloo^ or fiwncwoeta for vndemandins, d^cribins or ch««ct ert zl B g 

For example: to recognise a probkn and define it* character without prejudice to inquiry: to 
determine a useful way of totting and sub«lessifying information; to make an undentendeble 
report of what one experienced in a given situstion; to cUwify data, finding* or opiniona weine 
a gtvan classification schema. • 

1.2 DECODINO SIONIFICANCB: 

'JH^^SH^ * "? ** ? tf0fm « ,to ^ «"«•««. P«T»rt, directivt function* intentions, motives, purposes, 
(octal •ignirTcwca, vnleee, views, ruler, procedure!, criteria, or inferential relationships expreJun cooWtk^S 
communication aysttme, and. a. in ilanyage, eodi behaviors, drawings, numbers, gnpne.^leTZrU, S^S^SST 
For exsmpte: to detect end describe ■ person'* purpo.ee to setting a given quetlton; to appreciate 
the significance of a particular facial expression or gesture used in e given social situation; to 
discern the use of irony or rhetorical questions In debate; to interpret the data displayed or 
presented using a particular form of instrumentation. 

1 J CLARIFYING MEANING: 

• to paraphrase or make explicit, through stipulation, description, analogy or figurative expression, the contextual, conventional 
or intended meanings of words, ideas, concept*, statements, behaviors, drawings, numbers, signs, charts, graphs, symbols, rules, 

• to use stipulation, deacription, analogy or figurative expression to remove confusing, unintended vagueness or ambltuity. or 
to design a reasonable procedure for so doing. * " 

For example: to restate what a parson said using different words or expressions while preserving 
that person's Intended meanings; to find an example which helps explain something to someone; 
to develop a distinction which makea dear a conceptual difference or removes a troublesome 
ambiguity. 



Recently developed penological theories, such as the Ajxen-Ftohbind Theory of Reasoned Action, may afford new 
to access CT dispositions using objective letting instrumentation. 
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Cm m>l «*■ AMALVSI& To identify the iatMM and actual iafereatiel reletiooships among itatameats, questions, concepts, 
descriptions of other forms of reptaeaatation intended to express beliefs, judgments, experiences, reaeoee, information, or opinions. 

2.1 EXAMIMNO IDEAS: 

* 10 determine the rote varioue rxprtsiioos ptayorart intended to play to the context of argument, rtaeoaing or peieuasioa. 
9 to define um 

• to compute or contrast ideas, concepts, or statements* 

• to identify humor proMome aad determine their component porta, and atoo to identify the conceptual relatiooshlpe of those 
pans to each other end to the whole. 

Woe example? to Identify a phrase intended lo trigger a sympathetic emotional fmpam wMch 
mlfht induce an iiii ac i to agree with aa opinion; to eaamine closely ratatad proposals 
reading a gtteo problem and to determine their poin* of similarity at* 
complicated ai^iwat, to determine hoar it might babrokaa up into smaller, more manageable 
tasks; to defloa aa Attract concept 

12 DETECTING ARGUMENT* 

* given a eat of stateaaeata, d es cri p ti on s , question! or graphic reprtseatetions, to determine whether or not the set expresses, 
or is intended to express, a reason or teaeons in support of or contesting some claim, opinion or point of view, 

P6r example, ghm a paragraph, datermine whether a standard reading of that paragraph in the 
context of hoar and where it is published, would suggest thet it presents a daim as waH as a 
reason or reasons in support of that data; given a pa wags from a newspaper editorial, 
determine if the author of that pessage intended It aa an expression of reaeons for or against s 
given daim or opinion; given a commercial tonounccmeiu, identify any daime being advanced 
along with tba raaaons ntaaantad in their am most, 

2,3 ANALYZING ARGUMENTS: 

* given the expression of a reason or rsaaons intended to support or contest some claim, opinion or point of view, to identity 
and differentiate: (a) the intended main conduskm, (b) the premises and rsaaons advanced in support of the main conclusion, 
(c) further premises and reasons advanced aa backup or support for those premises and reasons intended as supporting the main 
conduskm, (d) additional u ne xpr essed elements of that reasoning, such as intermediary conclusions, unstated assumptions or 
presuppositions, (e) the overall structure of the argument or intended chain of reasoning, and (f) any items contained in the body 
of ex press i o ns being ex amined which are not intended to be taken as part of the reasoning being expressed or it* intended 
background. 

For example: given a brief argument, peragraph«eiaed argument, or a position paper on a 
controversial social issue, to identify the author's chief daim, the rsaaons and premises the 
author advances on behalf of that deha, the background information used to support those 
reasons or premises, and crucial assumptions ImpUdt In the author's reaming; gjwa several 
reasons or chains of rea s ons in eupport of n particular daim, to develop a graphic teptoseatotion 
which usefeOy characterizes the inferential flow of that reasoning. 



Com Skill #3. INTERPRETATION: To assess the credibility of statements or other representations which sre accounts or descriptions 
of a person's perception, experience, situation, judgment, belief, or opinion; and to assess the logical strength of the actual or intended 
inferential relationships among statements, descriptions, questions or other forms of representation. 

3.1 ASSESSING CLAIMS: 

• to recognize the factors relevant to assessing the degree of credibility to escribe to a source of information or opinion. 

• to assess the contextual relevance of questions, information, principles, rules or procedurel directions, 

9 to assess the acceptability, the level of confidence to place in the probability or truth of any given representation of an 

experience, situation, judgment, belief or opinion. 

P6r exar^r. to recognize the factors which make a person a credible witness regarding a given 
event Of edible authority on a given topic; to determine *f a given prindptc of conduct is 
applicable to deciding what to do in a given situation; to determine if a given claim is likely to 
be true or fatee baaed on what one knows or can reasonably find out 

i2 ASSESSING ARGUMENTS: , . , , 

• to judge whether the assumed acceptability of the premises of a given argument justify one's accepting es true (deductively 
certain), or very probably true (inductively justified), the expressed conduskm of that argument 

• to anticipate or to rsise questions or objections, and to assess whether these point to significant weakness in the argument being 
evaluated. 

• to determine whether aa argument relies on false or Joubtful assumptions or presuppositions and then to determine how 
crucially these affect its strength. 
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* to judge between reaaonabte end faOerioua inferences; 

^Tlitj^T** 1 *** * " i,T,n,Wrt *" Ump « ,0fti • v<«w tow.m deHnninin, «hc acceptability 

.^rST ( !^iL2T ttt ^i^l^^"** foJI<™aitherwitli ctmtaty or *th 
t^ZZ^ZSZXllT* ***** to check for fcfaatiflable farm*, 4*4 informal 
fallecoet; given m objection to m argument to evaluate tiM fogkal font of that objection: to 
eve!uete «heqe«Hty aadaptfkaWHty of aaalogtet argun***!^** ike logkJm^ 5 

r »^i<**K»^orh^ 
ntw^tamigAtlcadtonjcaUy^ 

CaBtagl #4, [NrERRNffiV To kS— tif> «»* fcure elements needed to drew laanoonbte cohesions; to form eoniecture. sad hv^h^. 
opinions, concepts, dcecriptlont, owastioas, or other tent of leptnnantatfon. 
4.1 QUERYING EVIDENCE 

" lUfSH?? 1, to J^^ '»*>"»■««» wkvMt to deciding it* acceptebiiHy, piauiibilify or relatiw: merits of • given alternative, 
SltoSiT*' ' h * ( * imk ' 0 ™*™ iat te ******* tormina pUu*b(e investigatory strategies for acquiring that 

For exempt* when attempting to dewJop a pertuaeive argument in aupport of ooa't opinion, to 
judge whet background information it M)ddba ^hawand todavatopapJan^S^ 
yield a dear answer at to whether or not each Inforaution lamiiab^aftarjtidfioi tl»ai<attain 
misting Information would ba germane in determining if a given opinion mora or test 
reasonable than a compering opinion, to plan a March which will reveal if that information it 

IVllllblCi 

42 CONJECTURINO ALTERNATIVES: 

' to formulate multiple aUamarivaa for resolving a problem, to postulate a aeries of tuppoeitiona regarding a question, to project 
alternative hypotheses regarding an event, to develop a variety of different plana to achieve eooeVeJ. 
' todrmvout pi^yppo^ theories, or belled 

For example: given a problem with technical, ethical or budgetary rantiflcarione, to devetopa set 
of options for addressing and resoJvieg that problem; given a set of priorities with which one 
may or may not agree, to project the dtfficultiss and the benefits which are Ukely to result if 
those priorities ait adopted in decision making. 

4.3 DRAWING CONCLUSIONS: 

• to apply appropriate modes of inference it. determining what position, opinion or point of view one should take on a ejven 
matter or issue, 

• given a set of statements, descriptions, queationa or other forms of representation, to educe, with the proper level of logical 
strength, their inferential relationships and the coneaquencee or the presuppositions which they support, warrant, imply or entail. 

• to employ successfully various subspecies of reasoning, at for example to reason analogically, arithmetically, dielectkaUy, 
ttitAtiilctily, etc. 

• to determine which of several possible conclusions Is moat strongly warranted or supported by the evidence at hand, or which 
should be rejected or regarded at teat plausible by the information given. 

For example: to carry out experiments and to apply appropri a te statistical inference techniques 
in order to confirm or dlsconfirm as empirical hypothesis; given a controversial issue to examine 
informed opinions, consider various opposing views and the reasons advanced for them, gather 
relevant information, and formulate one's own c o m fc l s re d opinion regarding that lame: to 
deduce a theorem from axioms using prescribed rules of inference. 

»• 

r nVr CJM Ski " **• RXP1 - ANATTOK ' ! Ta ***** <)* «*««« of one's reasoning; to Justify that reasoning in terms of the evidential, conceptual, 
kJS&i: nwihodotogkel, eriteriotogteal and contextual considerations upon which one's results were baaed; and to present one's masoning in the 
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• to product accurate ■tatcments, daacriptionc or rspreasatations of th« results of one's masoning activities mm to analyze, 
evaluate, later from, or mooter thoss reouHs. 

Vot ex emp l i : lo Nam oacl w w far hottng ■ gteo view, to write do— for oaea own tutnrn 
uae ootl carnal thiakmg Aon m tapomot or compter matte* to mm cod munch 
fl»dm|Kwcosmyotc%mMte»mrt j¥»MM 
opiaioa oo a mittcr of practical urgency. 

5.2 JUSTIFYING PROCEDURES: 

• to present the evidential, conceptual, mctaodoiofica!, criterioloffcal end contextual considCTatioas wfeka one uied in forming 
onc't intcrpretatioos, rn l y aa i , evaluation or mfceaame, to mat one aright accurately record, evaluate, describe or Juatily thorn 
proccatct to ooe'e 4V or to others, or co ss to remedy pctcatVcd defldeacicB in the general way ooa caccutce t how processes. 

For ammpte to hacp a log of the Maps tote^m«oittt|tlMoafhclOHoromuhpfobten 
or actamms procedure; to cxptta oaa% chafes of a paAkvteitatiatteJ tarn for parpoam of data 
anatyifc; to state the standards oaa tread to evuluctJogeDieceof Uteres to cxpieMfcw one 
oodaittaadi a hay coacapt what coa c tplaal cferity k crucial for fotthar program oa a gfcen 
pioMeavteiaowthmtterHmi»Mi^ 

satisfied; to report the etratcgy aatd ia ettcmptiag lo mate a decision la ■ reasonable way; to 
daajp^a graphic dhplay whfch mpramaai taa qaaatteUva or spatial iafocmatioa amd as 

5.3 PRESENTING ARGUMENTS: 

* to give raaaona for accepting tome daim. 

* to meat obJactiOM to the method, conccptualixatioas, evidence criteria or contextual appropriateneet of Inferential, analytical 
or evaluativa judgments. 

For example: to write a paper in which oae argues for a given position or policy; to anticipate 
and to respond to reasonable criticisms one might expect to be rated against one's political 
views; to identify and express evidence and counter-evidence intended as a dialectical 
contribution to onea own or another person'! thinking on a matter of deep personal 



Core Skill #6 , jpi P.ttFi Wi^TION:S df^^ monitor ooc'e cognitive activities, the elements used ia those activities, and the 

results educed, particularly by applying stills ia analysis and evaluation to one's own inferential judgment* with a view toward 
questioning, confirming, validating! or correcting either ooet Masoning or one's results. 

6.1 SELF-EXAMINATION: 

• to reflect on one's own reasoning and verify both the results produced and the correct application and execution of the cognitive 
skills twolved. 

• to make an objective and thoughtful mete-cognitive self-assessment of one's opinions and reasons for holding them 

• to judge the extent to which one's thinking is influenced by deficiencies in one's knowledge, or by stereotypes, prejudices, 
emotions or any other factors which constrain one* objectivity or rationality. 

• to reflect on one's natations, values, attitudes and Interests with a view toward deutmining that one has endeavored to be 
unbiased, fair-minded* ttmuugh, objective, respectful of the truth, reasonable, and rational In coming to one's analyses, 
interpretations, evaluations, inferences, or expressions. 

For example: to examine one's views on n controversial issue with sensitivity to the possible 
influences of one's personal bias or self-interest; to review one's methodology or calculations 
with a view to detecting mistaken applications or inadvertent arms; to reread sources to assure 
that one has not overlooked Important information; to identify and review the acceptability of 
the facts, opinions or assumptions one relied on in coming to a given point of view, to identify 
and review one's reasons and reasoning p r oce s se s in coming to a given conclusion. 

6.2 SELF-CORRECTION; 

• where self-examination reveals errors or deficiencies, to design reasonable procedures to remedy or correct, if possible, those 
mistakes and their causes. 

Air example: given a methodologkal mistske or factual deficiency in one's work, to revise that 
work so aa to correct the problem and then to determine If the revisions warrant changes in any 
position, findings, or opinions based thereon. 
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Review of Paul-Nosich Paper "A Proposal for the 
National Assessment, of Higher -Order Thinking at the 
Community College, College, and University Levtis" 

by 

Ronald K. Hambleton 
University of Massachusetts at Amherst 

The authors remind us that the level of success in developing higher- 
order thinking skills among higher education students seems to be as low as it 
appears to be among elementary and secondary students. One might expect (or 
predict) better results among higher education students because of the goals 
and curricula of higher education, but the authors claim that this is not so. 

I'm not an expert in either the skills that define what is meant by 
higher-order thinking skills for community college, college, and university 
students or how these skills might be measured in a national assessment. But, 
what is clear from the Paul-Nosich paper is that these authors are experts, 
and that they and their colleagues, over an extended period of time, have 
compiled an impressive amount of relevant material for this workshop. 

In Section 1 of the paper, the authors offer a set of 21 objectives for 
a process to assess higher-order thinking at the post -secondary school level. 
I don't feel qualified to critique this list myself. What I would like to see 
eventually is evidence that the various objectives of a testing system that 
the authors advance are widely accepted by the educational community and which 
objectives (perhaps all of them) are needed to meet the intentions of 
Objective 5 of Goal 5. 

In Section 2, the authors explain how their conception of critical 
thinking meets their 21 objectives (criteria) for an assessment system. The 
arguments they offer for the consistency of their conception of critical 
thinking with the 21 objectives was carefully prepared. Certainly, I could 
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